2019-10-01T01:59:19.705 INFO:root:teuthology version: 1.0.0-df92266 2019-10-01T01:59:19.706 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.bduxrB.tmp -- --verbose --lock --description rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml} --name kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574 2019-10-01T01:59:19.760 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-01T01:59:19.813 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574 branch: wip-kefu-testing-2019-09-30-1336 description: rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '4347574' kernel: kdb: true sha1: distro last_in_suite: false machine_type: mira name: kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2019-09-30-1336 ceph: conf: global: ms inject socket failures: 5000 ms type: async mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - overall HEALTH_ - \(MON_DOWN\) sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2019-09-30-1336 sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 suite: rados suite_branch: wip-kefu-testing-2019-09-30-1336 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master tube: mira verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.mira.11875 2019-10-01T01:59:19.813 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa; will attempt to use it 2019-10-01T01:59:19.814 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa/tasks 2019-10-01T01:59:19.814 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-01T01:59:19.816 INFO:teuthology.task.internal:Checking packages... 2019-10-01T01:59:19.829 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '2c0b2169f6303e11ab6bed736272e5e2d23ab027' 2019-10-01T01:59:19.830 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-01T01:59:19.830 INFO:teuthology.packaging:ref: None 2019-10-01T01:59:19.830 INFO:teuthology.packaging:tag: None 2019-10-01T01:59:19.830 INFO:teuthology.packaging:branch: wip-kefu-testing-2019-09-30-1336 2019-10-01T01:59:19.830 INFO:teuthology.packaging:sha1: 2c0b2169f6303e11ab6bed736272e5e2d23ab027 2019-10-01T01:59:19.830 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2019-09-30-1336 2019-10-01T01:59:19.983 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-5673.g2c0b216 2019-10-01T01:59:19.983 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-01T01:59:19.992 INFO:teuthology.task.internal:no buildpackages task found 2019-10-01T01:59:19.992 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-01T01:59:20.009 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-01T01:59:20.009 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-01T01:59:20.055 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574', 'machine_type': 'mira'} 2019-10-01T01:59:20.068 DEBUG:teuthology.lock.ops:locked ubuntu@mira063.front.sepia.ceph.com, ubuntu@mira064.front.sepia.ceph.com 2019-10-01T01:59:20.233 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-01T01:59:20.453 INFO:teuthology.provision.fog.mira064:Scheduling deploy of centos 7.6 2019-10-01T01:59:20.456 INFO:teuthology.provision.fog.mira063:Scheduling deploy of centos 7.6 2019-10-01T01:59:20.795 INFO:teuthology.orchestra.console:Power off mira063 2019-10-01T01:59:20.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-01T01:59:20.844 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:20.902 INFO:teuthology.orchestra.console:Power off mira064 2019-10-01T01:59:20.902 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-01T01:59:20.949 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:24.934 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:25.005 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:29.068 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:29.134 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:29.377 INFO:teuthology.orchestra.console:Power off for mira063 completed 2019-10-01T01:59:29.442 INFO:teuthology.orchestra.console:Power off for mira064 completed 2019-10-01T01:59:29.478 INFO:teuthology.orchestra.console:Power on mira063 2019-10-01T01:59:29.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-01T01:59:29.543 INFO:teuthology.orchestra.console:Power on mira064 2019-10-01T01:59:29.544 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-01T01:59:29.576 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:29.627 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:33.660 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:33.693 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:37.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:37.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-01T01:59:38.032 INFO:teuthology.orchestra.console:Power on for mira063 completed 2019-10-01T01:59:38.056 INFO:teuthology.orchestra.console:Power on for mira064 completed 2019-10-01T01:59:38.133 INFO:teuthology.provision.fog.mira063:Waiting for deploy to finish 2019-10-01T01:59:38.158 INFO:teuthology.provision.fog.mira064:Waiting for deploy to finish 2019-10-01T02:02:09.341 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:09.350 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:18.410 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:18.411 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:27.469 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:27.470 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:36.528 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:36.530 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:45.618 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:45.621 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:54.700 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:02:54.702 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:03:03.757 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:04:00.750 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:04:01.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:01.429 INFO:teuthology.orchestra.run.mira063:> hostname 2019-10-01T02:04:02.646 INFO:teuthology.orchestra.run.mira063.stdout:mira063.front.sepia.ceph.com 2019-10-01T02:04:02.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:02.646 INFO:teuthology.orchestra.run.mira063:> grep mira063.front.sepia.ceph.com /etc/hosts 2019-10-01T02:04:02.777 INFO:teuthology.orchestra.run.mira063.stdout:172.21.6.124 mira063.front.sepia.ceph.com mira063 2019-10-01T02:04:02.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:02.778 INFO:teuthology.orchestra.run.mira063:> sudo hostname mira063 2019-10-01T02:04:03.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:03.192 INFO:teuthology.orchestra.run.mira063:> sudo sed -i -e 's/mira063.front.sepia.ceph.com/mira063/g' /etc/hosts 2019-10-01T02:04:03.341 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:03.341 INFO:teuthology.orchestra.run.mira063:> sudo sed -i -e 's/172.21.6.124/172.21.6.124/g' /etc/hosts 2019-10-01T02:04:03.505 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:03.505 INFO:teuthology.orchestra.run.mira063:> sudo sed -i -e 's/mira063.front.sepia.ceph.com/mira063/g' /etc/hostname 2019-10-01T02:04:03.666 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:03.666 INFO:teuthology.orchestra.run.mira063:> sudo hostname mira063 2019-10-01T02:04:03.814 INFO:teuthology.provision.fog.mira063:Deploy complete! 2019-10-01T02:04:03.821 DEBUG:teuthology.parallel:result is None 2019-10-01T02:04:09.759 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:04:10.465 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:10.465 INFO:teuthology.orchestra.run.mira064:> hostname 2019-10-01T02:04:12.208 INFO:teuthology.orchestra.run.mira064.stdout:mira064.front.sepia.ceph.com 2019-10-01T02:04:12.208 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:12.208 INFO:teuthology.orchestra.run.mira064:> grep mira064.front.sepia.ceph.com /etc/hosts 2019-10-01T02:04:12.326 INFO:teuthology.orchestra.run.mira064.stdout:172.21.6.120 mira064.front.sepia.ceph.com mira064 2019-10-01T02:04:12.327 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:12.327 INFO:teuthology.orchestra.run.mira064:> sudo hostname mira064 2019-10-01T02:04:12.767 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:12.767 INFO:teuthology.orchestra.run.mira064:> sudo sed -i -e 's/mira064.front.sepia.ceph.com/mira064/g' /etc/hosts 2019-10-01T02:04:12.939 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:12.940 INFO:teuthology.orchestra.run.mira064:> sudo sed -i -e 's/172.21.6.120/172.21.6.120/g' /etc/hosts 2019-10-01T02:04:13.109 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:13.109 INFO:teuthology.orchestra.run.mira064:> sudo sed -i -e 's/mira064.front.sepia.ceph.com/mira064/g' /etc/hostname 2019-10-01T02:04:13.280 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:13.280 INFO:teuthology.orchestra.run.mira064:> sudo hostname mira064 2019-10-01T02:04:13.446 INFO:teuthology.provision.fog.mira064:Deploy complete! 2019-10-01T02:04:13.448 DEBUG:teuthology.parallel:result is None 2019-10-01T02:04:13.686 INFO:teuthology.lock.keys:Checking mira063.front.sepia.ceph.com 2019-10-01T02:04:13.687 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-01T02:04:13.702 INFO:teuthology.lock.keys:Checking mira064.front.sepia.ceph.com 2019-10-01T02:04:13.702 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-01T02:04:13.719 INFO:teuthology.task.internal.lock_machines:2 mira machines locked this try, 2/2 locked so far 2019-10-01T02:04:13.750 INFO:teuthology.task.internal.lock_machines:Locked targets: mira063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC0syhJayVLaGcvkxkrH7lkHRQ+9cOZOrjrmRflouJyaEdL6Qo0rCPCL+2XUeLe7iZbSquaSvxF/shLm7hQWBC2wNMPB+BY//Xe0v9kcN7W7IJ7TEkOuQBHx6mWftT5ncTcmsyZtixRos+ejLoQCqJdvNVIAaqHExwvgvFPFRJGloz7ceptcY90+IkonAxuSvtO3vmeEvTb1q5bDZpd4mveI1+XlTLyFhMuLziYImO5UBvStnBGbgRV6BaNQcNvRF59VF25S/nHBQtm3FUGcF5mqdARwJq95F6v7ZP0FVdOyG5JZ0j/XaIXeQysaG3AFNNUCggjNHZKhMxX1+L4KFf mira064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE+Ux0+98wzMAe6H0MZQA31ni2vyP0dNZ8EZAFJBqpq0/AITvn7JrRa3WSI6FItIYUdS3tzogVRnnQCjq4/I4CWwxw9pFpYSc0Ikx3r8TVo5qqqrSGkcSRo6DCFZe/GVQZaAAhclynmA/1oPxr62FD7ohW9oWlHOZE/vtcqxKiTjxXTrsxE31qRh8VA3V8x/5/ZijfORMjqQKWKhkJ1oPOOGqi81G7cTatOnA4Ri1PCB8UO34p5MX9ESSUFqaw+BPql5olSoenD3aIEeCCLofuNQ85Gh9nwQSKG+Q08Fb9HSzD3QRBl9JE0j65ZUr8Z6Pf37qv6wg9NNGXTm2Coq69 2019-10-01T02:04:13.750 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-01T02:04:13.797 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-01T02:04:13.805 INFO:teuthology.task.internal:Saving configuration 2019-10-01T02:04:13.844 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-01T02:04:13.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-01T02:04:13.869 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-01 01:59:20.061316', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': u'00:25:90:32:bf:28', u'name': u'mira063.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'mira', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC0syhJayVLaGcvkxkrH7lkHRQ+9cOZOrjrmRflouJyaEdL6Qo0rCPCL+2XUeLe7iZbSquaSvxF/shLm7hQWBC2wNMPB+BY//Xe0v9kcN7W7IJ7TEkOuQBHx6mWftT5ncTcmsyZtixRos+ejLoQCqJdvNVIAaqHExwvgvFPFRJGloz7ceptcY90+IkonAxuSvtO3vmeEvTb1q5bDZpd4mveI1+XlTLyFhMuLziYImO5UBvStnBGbgRV6BaNQcNvRF59VF25S/nHBQtm3FUGcF5mqdARwJq95F6v7ZP0FVdOyG5JZ0j/XaIXeQysaG3AFNNUCggjNHZKhMxX1+L4KFf', u'description': u'/home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574'} 2019-10-01T02:04:13.885 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-01 01:59:20.061337', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': u'00:25:90:09:e2:d4', u'name': u'mira064.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'mira', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE+Ux0+98wzMAe6H0MZQA31ni2vyP0dNZ8EZAFJBqpq0/AITvn7JrRa3WSI6FItIYUdS3tzogVRnnQCjq4/I4CWwxw9pFpYSc0Ikx3r8TVo5qqqrSGkcSRo6DCFZe/GVQZaAAhclynmA/1oPxr62FD7ohW9oWlHOZE/vtcqxKiTjxXTrsxE31qRh8VA3V8x/5/ZijfORMjqQKWKhkJ1oPOOGqi81G7cTatOnA4Ri1PCB8UO34p5MX9ESSUFqaw+BPql5olSoenD3aIEeCCLofuNQ85Gh9nwQSKG+Q08Fb9HSzD3QRBl9JE0j65ZUr8Z6Pf37qv6wg9NNGXTm2Coq69', u'description': u'/home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574'} 2019-10-01T02:04:13.885 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-01T02:04:13.894 INFO:teuthology.task.internal:roles: ubuntu@mira063.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2019-10-01T02:04:13.894 INFO:teuthology.task.internal:roles: ubuntu@mira064.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2019-10-01T02:04:13.895 INFO:teuthology.run_tasks:Running task console_log... 2019-10-01T02:04:14.074 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-01T02:04:14.075 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-01T02:04:14.082 INFO:teuthology.task.internal:Opening connections... 2019-10-01T02:04:14.083 DEBUG:teuthology.task.internal:connecting to ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:04:14.084 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:04:14.223 DEBUG:teuthology.task.internal:connecting to ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:04:14.223 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:04:14.373 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-01T02:04:14.866 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:14.866 INFO:teuthology.orchestra.run.mira063:> uname -m 2019-10-01T02:04:15.018 INFO:teuthology.orchestra.run.mira063.stdout:x86_64 2019-10-01T02:04:15.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:15.019 INFO:teuthology.orchestra.run.mira063:> cat /etc/os-release 2019-10-01T02:04:15.131 INFO:teuthology.orchestra.run.mira063.stdout:NAME="CentOS Linux" 2019-10-01T02:04:15.131 INFO:teuthology.orchestra.run.mira063.stdout:VERSION="7 (Core)" 2019-10-01T02:04:15.131 INFO:teuthology.orchestra.run.mira063.stdout:ID="centos" 2019-10-01T02:04:15.131 INFO:teuthology.orchestra.run.mira063.stdout:ID_LIKE="rhel fedora" 2019-10-01T02:04:15.131 INFO:teuthology.orchestra.run.mira063.stdout:VERSION_ID="7" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:ANSI_COLOR="0;31" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:HOME_URL="https://www.centos.org/" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-10-01T02:04:15.132 INFO:teuthology.orchestra.run.mira063.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-10-01T02:04:15.133 INFO:teuthology.orchestra.run.mira063.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-10-01T02:04:15.133 INFO:teuthology.orchestra.run.mira063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-10-01T02:04:15.133 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:15.134 INFO:teuthology.lock.ops:Updating mira063.front.sepia.ceph.com on lock server 2019-10-01T02:04:15.147 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:15.147 INFO:teuthology.orchestra.run.mira064:> uname -m 2019-10-01T02:04:15.320 INFO:teuthology.orchestra.run.mira064.stdout:x86_64 2019-10-01T02:04:15.321 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:15.321 INFO:teuthology.orchestra.run.mira064:> cat /etc/os-release 2019-10-01T02:04:15.437 INFO:teuthology.orchestra.run.mira064.stdout:NAME="CentOS Linux" 2019-10-01T02:04:15.438 INFO:teuthology.orchestra.run.mira064.stdout:VERSION="7 (Core)" 2019-10-01T02:04:15.438 INFO:teuthology.orchestra.run.mira064.stdout:ID="centos" 2019-10-01T02:04:15.438 INFO:teuthology.orchestra.run.mira064.stdout:ID_LIKE="rhel fedora" 2019-10-01T02:04:15.438 INFO:teuthology.orchestra.run.mira064.stdout:VERSION_ID="7" 2019-10-01T02:04:15.439 INFO:teuthology.orchestra.run.mira064.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-10-01T02:04:15.439 INFO:teuthology.orchestra.run.mira064.stdout:ANSI_COLOR="0;31" 2019-10-01T02:04:15.439 INFO:teuthology.orchestra.run.mira064.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-10-01T02:04:15.439 INFO:teuthology.orchestra.run.mira064.stdout:HOME_URL="https://www.centos.org/" 2019-10-01T02:04:15.440 INFO:teuthology.orchestra.run.mira064.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-10-01T02:04:15.440 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:04:15.440 INFO:teuthology.orchestra.run.mira064.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-10-01T02:04:15.448 INFO:teuthology.orchestra.run.mira064.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-10-01T02:04:15.449 INFO:teuthology.orchestra.run.mira064.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-10-01T02:04:15.449 INFO:teuthology.orchestra.run.mira064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-10-01T02:04:15.449 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:04:15.449 INFO:teuthology.lock.ops:Updating mira064.front.sepia.ceph.com on lock server 2019-10-01T02:04:15.477 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-01T02:04:15.533 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-01T02:04:15.554 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-01T02:04:15.555 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:15.555 INFO:teuthology.orchestra.run.mira063:> test '!' -e /home/ubuntu/cephtest 2019-10-01T02:04:15.566 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:15.566 INFO:teuthology.orchestra.run.mira064:> test '!' -e /home/ubuntu/cephtest 2019-10-01T02:04:15.680 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-01T02:04:15.695 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-01T02:04:15.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:15.696 INFO:teuthology.orchestra.run.mira063:> test -z $(ls -A /var/lib/ceph) 2019-10-01T02:04:15.703 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:15.703 INFO:teuthology.orchestra.run.mira064:> test -z $(ls -A /var/lib/ceph) 2019-10-01T02:04:15.817 INFO:teuthology.orchestra.run.mira063.stderr:ls: cannot access /var/lib/ceph: No such file or directory 2019-10-01T02:04:15.818 INFO:teuthology.orchestra.run.mira064.stderr:ls: cannot access /var/lib/ceph: No such file or directory 2019-10-01T02:04:15.820 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-01T02:04:15.931 INFO:teuthology.run_tasks:Running task kernel... 2019-10-01T02:04:16.067 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2019-10-01T02:04:16.067 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-01T02:04:16.068 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:04:16.068 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:16.068 INFO:teuthology.orchestra.run.mira063:> sudo package-cleanup -y --oldkernels 2019-10-01T02:04:19.432 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:04:25.759 INFO:teuthology.orchestra.run.mira063.stdout:No old kernels to remove 2019-10-01T02:04:25.786 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:04:25.786 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:04:25.787 INFO:teuthology.orchestra.run.mira064:> sudo package-cleanup -y --oldkernels 2019-10-01T02:04:30.636 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:04:35.694 INFO:teuthology.orchestra.run.mira064.stdout:No old kernels to remove 2019-10-01T02:04:35.713 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:35.713 INFO:teuthology.orchestra.run.mira063:> uname -r 2019-10-01T02:04:35.845 INFO:teuthology.orchestra.run.mira063.stdout:3.10.0-957.21.3.el7.x86_64 2019-10-01T02:04:35.845 INFO:teuthology.task.kernel:Running kernel on mira063: 3.10.0-957.21.3.el7.x86_64 2019-10-01T02:04:35.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:04:35.846 INFO:teuthology.orchestra.run.mira063:> sudo yum install -y kernel 2019-10-01T02:04:36.274 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:04:36.405 INFO:teuthology.orchestra.run.mira063.stdout:Determining fastest mirrors 2019-10-01T02:04:36.678 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.math.princeton.edu 2019-10-01T02:04:36.679 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:04:36.680 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirrors.mit.edu 2019-10-01T02:04:36.681 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.ash.fastserv.com 2019-10-01T02:04:41.061 INFO:teuthology.orchestra.run.mira063.stdout:1 packages excluded due to repository priority protections 2019-10-01T02:04:41.690 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:04:41.691 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:04:41.691 INFO:teuthology.orchestra.run.mira063.stdout:---> Package kernel.x86_64 0:3.10.0-1062.1.1.el7 will be installed 2019-10-01T02:04:42.567 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:04:42.807 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:04:42.807 INFO:teuthology.orchestra.run.mira063.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-10-01T02:04:42.841 INFO:teuthology.orchestra.run.mira063.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-10-01T02:04:47.162 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:04:50.938 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:50.938 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:04:50.965 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:50.965 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:04:50.965 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:04:50.965 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout: kernel x86_64 3.10.0-1062.1.1.el7 updates 50 M 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout:Updating for dependencies: 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:50.966 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:04:50.967 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:04:50.967 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package 2019-10-01T02:04:50.967 INFO:teuthology.orchestra.run.mira063.stdout:Upgrade ( 1 Dependent package) 2019-10-01T02:04:50.967 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:04:50.967 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 123 M 2019-10-01T02:04:50.968 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:04:50.968 INFO:teuthology.orchestra.run.mira063.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-01T02:04:52.985 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:04:52.986 INFO:teuthology.orchestra.run.mira063.stdout:Total 61 MB/s | 123 MB 00:02 2019-10-01T02:04:53.089 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:04:53.473 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:04:56.926 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:04:56.927 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:05:19.433 INFO:teuthology.orchestra.run.mira063.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-10-01T02:05:27.331 INFO:teuthology.orchestra.run.mira063.stdout: Installing : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-10-01T02:06:22.331 INFO:teuthology.orchestra.run.mira063.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-01T02:06:22.498 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-10-01T02:06:22.574 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-10-01T02:06:22.790 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-01T02:06:22.790 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout: kernel.x86_64 0:3.10.0-1062.1.1.el7 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Updated: 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:06:22.791 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:06:22.907 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:06:22.908 INFO:teuthology.orchestra.run.mira063:> rpm -q kernel --last | head -n 1 2019-10-01T02:06:23.576 INFO:teuthology.orchestra.run.mira063.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Tue 01 Oct 2019 02:05:19 AM UTC 2019-10-01T02:06:23.580 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:06:23.580 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:06:23.580 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:06:23.580 INFO:teuthology.orchestra.run.mira064:> uname -r 2019-10-01T02:06:23.742 INFO:teuthology.orchestra.run.mira064.stdout:3.10.0-957.21.3.el7.x86_64 2019-10-01T02:06:23.743 INFO:teuthology.task.kernel:Running kernel on mira064: 3.10.0-957.21.3.el7.x86_64 2019-10-01T02:06:23.743 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:06:23.743 INFO:teuthology.orchestra.run.mira064:> sudo yum install -y kernel 2019-10-01T02:06:24.418 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:06:24.490 INFO:teuthology.orchestra.run.mira064.stdout:Determining fastest mirrors 2019-10-01T02:06:24.729 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.facebook.net 2019-10-01T02:06:24.730 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:06:24.730 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirrors.mit.edu 2019-10-01T02:06:24.731 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.ash.fastserv.com 2019-10-01T02:06:30.401 INFO:teuthology.orchestra.run.mira064.stdout:1 packages excluded due to repository priority protections 2019-10-01T02:06:31.030 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:06:31.031 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:06:31.031 INFO:teuthology.orchestra.run.mira064.stdout:---> Package kernel.x86_64 0:3.10.0-1062.1.1.el7 will be installed 2019-10-01T02:06:31.914 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:06:32.334 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:06:32.334 INFO:teuthology.orchestra.run.mira064.stdout:---> Package linux-firmware.noarch 0:20180911-69.git85c5d90.el7 will be updated 2019-10-01T02:06:32.474 INFO:teuthology.orchestra.run.mira064.stdout:---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be an update 2019-10-01T02:06:43.219 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:06:53.232 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:06:53.232 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:06:53.390 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:06:53.390 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:06:53.390 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:06:53.390 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:06:53.390 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout: kernel x86_64 3.10.0-1062.1.1.el7 updates 50 M 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout:Updating for dependencies: 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout: linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:06:53.391 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package 2019-10-01T02:06:53.392 INFO:teuthology.orchestra.run.mira064.stdout:Upgrade ( 1 Dependent package) 2019-10-01T02:06:53.392 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:06:53.392 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 123 M 2019-10-01T02:06:53.392 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:06:53.393 INFO:teuthology.orchestra.run.mira064.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-01T02:06:55.389 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:06:55.389 INFO:teuthology.orchestra.run.mira064.stdout:Total 62 MB/s | 123 MB 00:01 2019-10-01T02:06:55.506 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:06:56.054 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:07:04.761 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:07:04.761 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:07:42.010 INFO:teuthology.orchestra.run.mira064.stdout: Updating : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-10-01T02:07:47.299 INFO:teuthology.orchestra.run.mira064.stdout: Installing : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-10-01T02:08:34.773 INFO:teuthology.orchestra.run.mira064.stdout: Cleanup : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-01T02:08:34.975 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : linux-firmware-20190429-72.gitddde598.el7.noarch 1/3 2019-10-01T02:08:35.051 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : kernel-3.10.0-1062.1.1.el7.x86_64 2/3 2019-10-01T02:08:35.259 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : linux-firmware-20180911-69.git85c5d90.el7.noarch 3/3 2019-10-01T02:08:35.259 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:08:35.259 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:08:35.259 INFO:teuthology.orchestra.run.mira064.stdout: kernel.x86_64 0:3.10.0-1062.1.1.el7 2019-10-01T02:08:35.259 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:08:35.260 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Updated: 2019-10-01T02:08:35.260 INFO:teuthology.orchestra.run.mira064.stdout: linux-firmware.noarch 0:20190429-72.gitddde598.el7 2019-10-01T02:08:35.260 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:08:35.260 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:08:35.370 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:35.370 INFO:teuthology.orchestra.run.mira064:> rpm -q kernel --last | head -n 1 2019-10-01T02:08:35.842 INFO:teuthology.orchestra.run.mira064.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Tue 01 Oct 2019 02:07:42 AM UTC 2019-10-01T02:08:35.843 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:35.844 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-957.21.3.el7.x86_64 Expected: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:35.844 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-10-01T02:08:35.844 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-01T02:08:35.844 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-01T02:08:35.844 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-10-01T02:08:35.844 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@mira063.front.sepia.ceph.com, path=None, version=distro) 2019-10-01T02:08:35.845 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:35.845 INFO:teuthology.orchestra.run.mira063:> rpm -q kernel --last | head -n 1 2019-10-01T02:08:36.348 INFO:teuthology.orchestra.run.mira063.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Tue 01 Oct 2019 02:05:19 AM UTC 2019-10-01T02:08:36.351 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:36.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:36.351 INFO:teuthology.orchestra.run.mira063:> sudo rpm -qi grub2 2019-10-01T02:08:36.588 INFO:teuthology.orchestra.run.mira063.stdout:Name : grub2 2019-10-01T02:08:36.588 INFO:teuthology.orchestra.run.mira063.stdout:Epoch : 1 2019-10-01T02:08:36.588 INFO:teuthology.orchestra.run.mira063.stdout:Version : 2.02 2019-10-01T02:08:36.588 INFO:teuthology.orchestra.run.mira063.stdout:Release : 0.76.el7.centos 2019-10-01T02:08:36.588 INFO:teuthology.orchestra.run.mira063.stdout:Architecture: x86_64 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Install Date: Fri 12 Jul 2019 03:03:58 PM UTC 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Group : System Environment/Base 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Size : 0 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:License : GPLv3+ 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-10-01T02:08:36.589 INFO:teuthology.orchestra.run.mira063.stdout:Build Host : kbuilder.bsys.centos.org 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:Relocations : (not relocatable) 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:Packager : CentOS BuildSystem 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:Vendor : CentOS 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:URL : http://www.gnu.org/software/grub/ 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout:Description : 2019-10-01T02:08:36.590 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:08:36.591 INFO:teuthology.orchestra.run.mira063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-10-01T02:08:36.591 INFO:teuthology.orchestra.run.mira063.stdout:customizable bootloader with modular architecture. It supports a rich 2019-10-01T02:08:36.591 INFO:teuthology.orchestra.run.mira063.stdout:variety of kernel formats, file systems, computer architectures and 2019-10-01T02:08:36.591 INFO:teuthology.orchestra.run.mira063.stdout:hardware devices. 2019-10-01T02:08:36.592 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-10-01T02:08:36.592 INFO:teuthology.task.kernel:Updating grub on mira063 to boot 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:36.592 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:36.592 INFO:teuthology.orchestra.run.mira063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-10-01T02:08:37.157 INFO:teuthology.orchestra.run.mira063.stderr:Generating grub configuration file ... 2019-10-01T02:08:37.348 INFO:teuthology.orchestra.run.mira063.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:37.375 INFO:teuthology.orchestra.run.mira063.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.1.1.el7.x86_64.img 2019-10-01T02:08:37.894 INFO:teuthology.orchestra.run.mira063.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 2019-10-01T02:08:37.922 INFO:teuthology.orchestra.run.mira063.stderr:Found initrd image: /boot/initramfs-3.10.0-957.21.3.el7.x86_64.img 2019-10-01T02:08:37.993 INFO:teuthology.orchestra.run.mira063.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-10-01T02:08:38.026 INFO:teuthology.orchestra.run.mira063.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-10-01T02:08:38.089 INFO:teuthology.orchestra.run.mira063.stderr:Found linux image: /boot/vmlinuz-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0 2019-10-01T02:08:38.111 INFO:teuthology.orchestra.run.mira063.stderr:Found initrd image: /boot/initramfs-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0.img 2019-10-01T02:08:39.348 INFO:teuthology.orchestra.run.mira063.stderr:done 2019-10-01T02:08:39.352 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:39.352 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:08:39.472 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.yGhVmI1Kl4 2019-10-01T02:08:39.473 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:39.473 INFO:teuthology.orchestra.run.mira063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yGhVmI1Kl4 2019-10-01T02:08:39.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:39.635 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.yGhVmI1Kl4 2019-10-01T02:08:39.932 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.yGhVmI1Kl4 is 6KB 2019-10-01T02:08:40.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:40.061 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.yGhVmI1Kl4 2019-10-01T02:08:40.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:40.169 INFO:teuthology.orchestra.run.mira063:> sudo grub2-set-default 0 2019-10-01T02:08:40.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:08:40.346 INFO:teuthology.orchestra.run.mira063:> sudo shutdown -r now 2019-10-01T02:08:40.352 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-10-01T02:08:40.352 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@mira064.front.sepia.ceph.com, path=None, version=distro) 2019-10-01T02:08:40.352 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:40.352 INFO:teuthology.orchestra.run.mira064:> rpm -q kernel --last | head -n 1 2019-10-01T02:08:40.843 INFO:teuthology.orchestra.run.mira064.stdout:kernel-3.10.0-1062.1.1.el7.x86_64 Tue 01 Oct 2019 02:07:42 AM UTC 2019-10-01T02:08:40.846 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:40.847 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:40.847 INFO:teuthology.orchestra.run.mira064:> sudo rpm -qi grub2 2019-10-01T02:08:41.059 INFO:teuthology.orchestra.run.mira064.stdout:Name : grub2 2019-10-01T02:08:41.059 INFO:teuthology.orchestra.run.mira064.stdout:Epoch : 1 2019-10-01T02:08:41.059 INFO:teuthology.orchestra.run.mira064.stdout:Version : 2.02 2019-10-01T02:08:41.059 INFO:teuthology.orchestra.run.mira064.stdout:Release : 0.76.el7.centos 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Architecture: x86_64 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Install Date: Fri 12 Jul 2019 03:03:58 PM UTC 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Group : System Environment/Base 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Size : 0 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:License : GPLv3+ 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-10-01T02:08:41.060 INFO:teuthology.orchestra.run.mira064.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Build Host : kbuilder.bsys.centos.org 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Relocations : (not relocatable) 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Packager : CentOS BuildSystem 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Vendor : CentOS 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:URL : http://www.gnu.org/software/grub/ 2019-10-01T02:08:41.061 INFO:teuthology.orchestra.run.mira064.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout:Description : 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout:customizable bootloader with modular architecture. It supports a rich 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout:variety of kernel formats, file systems, computer architectures and 2019-10-01T02:08:41.062 INFO:teuthology.orchestra.run.mira064.stdout:hardware devices. 2019-10-01T02:08:41.063 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-10-01T02:08:41.063 INFO:teuthology.task.kernel:Updating grub on mira064 to boot 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:41.064 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:41.064 INFO:teuthology.orchestra.run.mira064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-10-01T02:08:41.637 INFO:teuthology.orchestra.run.mira064.stderr:Generating grub configuration file ... 2019-10-01T02:08:41.832 INFO:teuthology.orchestra.run.mira064.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:08:41.860 INFO:teuthology.orchestra.run.mira064.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.1.1.el7.x86_64.img 2019-10-01T02:08:42.376 INFO:teuthology.orchestra.run.mira064.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 2019-10-01T02:08:42.419 INFO:teuthology.orchestra.run.mira064.stderr:Found initrd image: /boot/initramfs-3.10.0-957.21.3.el7.x86_64.img 2019-10-01T02:08:42.489 INFO:teuthology.orchestra.run.mira064.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-10-01T02:08:42.522 INFO:teuthology.orchestra.run.mira064.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-10-01T02:08:42.584 INFO:teuthology.orchestra.run.mira064.stderr:Found linux image: /boot/vmlinuz-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0 2019-10-01T02:08:42.605 INFO:teuthology.orchestra.run.mira064.stderr:Found initrd image: /boot/initramfs-0-rescue-ed19c4d1a216479b9bcb22a6fb70d4b0.img 2019-10-01T02:08:43.852 INFO:teuthology.orchestra.run.mira064.stderr:done 2019-10-01T02:08:43.856 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:43.856 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:08:43.980 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.h1AESvekg5 2019-10-01T02:08:43.980 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:43.980 INFO:teuthology.orchestra.run.mira064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h1AESvekg5 2019-10-01T02:08:44.136 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:44.136 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.h1AESvekg5 2019-10-01T02:08:44.412 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.h1AESvekg5 is 6KB 2019-10-01T02:08:44.528 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:44.528 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.h1AESvekg5 2019-10-01T02:08:44.634 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:44.634 INFO:teuthology.orchestra.run.mira064:> sudo grub2-set-default 0 2019-10-01T02:08:44.836 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:08:44.836 INFO:teuthology.orchestra.run.mira064:> sudo shutdown -r now 2019-10-01T02:08:44.847 INFO:teuthology.misc:Re-opening connections... 2019-10-01T02:08:44.847 INFO:teuthology.misc:trying to connect to ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:08:44.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:08:44.847 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:09:44.860 DEBUG:teuthology.orchestra.remote:timed out 2019-10-01T02:09:44.861 INFO:teuthology.misc:trying to connect to ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:09:44.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:09:44.862 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:10:44.863 DEBUG:teuthology.orchestra.remote:timed out 2019-10-01T02:10:44.864 DEBUG:teuthology.misc:waited 120.017023087 2019-10-01T02:10:45.864 INFO:teuthology.misc:trying to connect to ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:10:45.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:10:45.865 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:10:46.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:46.001 INFO:teuthology.orchestra.run.mira063:> true 2019-10-01T02:10:46.468 DEBUG:teuthology.misc:waited 121.621237993 2019-10-01T02:10:47.469 INFO:teuthology.misc:trying to connect to ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:10:47.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:10:47.470 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:10:47.582 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:47.582 INFO:teuthology.orchestra.run.mira064:> true 2019-10-01T02:10:47.982 DEBUG:teuthology.misc:waited 123.135339022 2019-10-01T02:10:48.984 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-10-01T02:10:48.984 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.1.1.el7.x86_64"... 2019-10-01T02:10:48.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:48.985 INFO:teuthology.orchestra.run.mira063:> uname -r 2019-10-01T02:10:49.117 INFO:teuthology.orchestra.run.mira063.stdout:3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:10:49.117 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:10:49.117 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-01T02:10:49.117 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-10-01T02:10:49.117 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.1.1.el7.x86_64"... 2019-10-01T02:10:49.118 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:49.118 INFO:teuthology.orchestra.run.mira064:> uname -r 2019-10-01T02:10:49.236 INFO:teuthology.orchestra.run.mira064.stdout:3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:10:49.237 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.1.1.el7.x86_64 2019-10-01T02:10:49.237 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-01T02:10:50.237 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-10-01T02:10:50.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:50.238 INFO:teuthology.orchestra.run.mira063:> echo ttyS2 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-01T02:10:50.425 INFO:teuthology.orchestra.run.mira063.stdout:ttyS2 2019-10-01T02:10:50.441 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-10-01T02:10:50.441 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:50.441 INFO:teuthology.orchestra.run.mira064:> echo ttyS2 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-01T02:10:50.639 INFO:teuthology.orchestra.run.mira064.stdout:ttyS2 2019-10-01T02:10:50.652 INFO:teuthology.run_tasks:Running task internal.base... 2019-10-01T02:10:50.659 INFO:teuthology.task.internal:Creating test directory... 2019-10-01T02:10:50.660 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:50.660 INFO:teuthology.orchestra.run.mira063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-01T02:10:50.665 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:50.666 INFO:teuthology.orchestra.run.mira064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-01T02:10:50.773 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-10-01T02:10:50.784 INFO:teuthology.run_tasks:Running task internal.archive... 2019-10-01T02:10:50.793 INFO:teuthology.task.internal:Creating archive directory... 2019-10-01T02:10:50.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:50.793 INFO:teuthology.orchestra.run.mira063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-01T02:10:50.800 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:50.800 INFO:teuthology.orchestra.run.mira064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-01T02:10:50.926 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-10-01T02:10:50.937 INFO:teuthology.task.internal:Enabling coredump saving... 2019-10-01T02:10:50.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:50.938 INFO:teuthology.orchestra.run.mira063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-01T02:10:50.945 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:50.945 INFO:teuthology.orchestra.run.mira064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-01T02:10:51.095 INFO:teuthology.orchestra.run.mira063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-01T02:10:51.112 INFO:teuthology.orchestra.run.mira064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-01T02:10:51.115 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-10-01T02:10:51.124 INFO:teuthology.task.internal:Configuring sudo... 2019-10-01T02:10:51.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.124 INFO:teuthology.orchestra.run.mira063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-01T02:10:51.142 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:51.142 INFO:teuthology.orchestra.run.mira064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-01T02:10:51.313 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-10-01T02:10:51.329 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-10-01T02:10:51.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.329 INFO:teuthology.orchestra.run.mira063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-01T02:10:51.335 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:51.336 INFO:teuthology.orchestra.run.mira064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-01T02:10:51.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.453 INFO:teuthology.orchestra.run.mira063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-01T02:10:51.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.593 INFO:teuthology.orchestra.run.mira063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-01T02:10:51.757 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.757 INFO:teuthology.orchestra.run.mira063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-01T02:10:51.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:51.892 INFO:teuthology.orchestra.run.mira063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-01T02:10:52.050 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:52.050 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-01T02:10:52.215 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.216 INFO:teuthology.orchestra.run.mira064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-01T02:10:52.368 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.368 INFO:teuthology.orchestra.run.mira064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-01T02:10:52.534 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.535 INFO:teuthology.orchestra.run.mira064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-01T02:10:52.630 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.630 INFO:teuthology.orchestra.run.mira064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-01T02:10:52.714 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.714 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-01T02:10:52.797 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:52.798 INFO:teuthology.orchestra.run.mira063:> sudo service rsyslog restart 2019-10-01T02:10:52.804 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:52.804 INFO:teuthology.orchestra.run.mira064:> sudo service rsyslog restart 2019-10-01T02:10:52.981 INFO:teuthology.orchestra.run.mira064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-10-01T02:10:52.997 INFO:teuthology.orchestra.run.mira063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-10-01T02:10:53.039 INFO:teuthology.run_tasks:Running task internal.timer... 2019-10-01T02:10:53.053 INFO:teuthology.task.internal:Starting timer... 2019-10-01T02:10:53.053 INFO:teuthology.run_tasks:Running task pcp... 2019-10-01T02:10:53.109 INFO:teuthology.run_tasks:Running task selinux... 2019-10-01T02:10:53.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:53.119 INFO:teuthology.orchestra.run.mira063:> sudo service auditd rotate 2019-10-01T02:10:53.347 INFO:teuthology.orchestra.run.mira063.stdout:Rotating logs: [ OK ] 2019-10-01T02:10:53.351 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:53.351 INFO:teuthology.orchestra.run.mira064:> sudo service auditd rotate 2019-10-01T02:10:53.630 INFO:teuthology.orchestra.run.mira064.stdout:Rotating logs: [ OK ] 2019-10-01T02:10:53.636 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-10-01T02:10:53.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:53.636 INFO:teuthology.orchestra.run.mira063:> /usr/sbin/getenforce 2019-10-01T02:10:53.757 INFO:teuthology.orchestra.run.mira063.stdout:Permissive 2019-10-01T02:10:53.758 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:53.758 INFO:teuthology.orchestra.run.mira064:> /usr/sbin/getenforce 2019-10-01T02:10:53.877 INFO:teuthology.orchestra.run.mira064.stdout:Permissive 2019-10-01T02:10:53.877 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@mira063.front.sepia.ceph.com': 'permissive', 'ubuntu@mira064.front.sepia.ceph.com': 'permissive'} 2019-10-01T02:10:53.878 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:53.878 INFO:teuthology.orchestra.run.mira063:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-10-01T02:10:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-01T02:10:54.028 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:54.028 INFO:teuthology.orchestra.run.mira064:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-10-01T02:10:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-01T02:10:54.190 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-10-01T02:10:54.191 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:10:54.191 INFO:teuthology.orchestra.run.mira063:> sudo /usr/sbin/setenforce permissive 2019-10-01T02:10:54.347 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:10:54.347 INFO:teuthology.orchestra.run.mira064:> sudo /usr/sbin/setenforce permissive 2019-10-01T02:10:54.539 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-10-01T02:10:54.551 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2019-10-01T02:10:54.551 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch master 2019-10-01T02:10:54.589 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'set_python_path.yml'}, {'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-10-01T02:10:54.592 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit mira063.front.sepia.ceph.com,mira064.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-10-01T02:10:54.998 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-10-01T02:10:58.704 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-01T02:10:58.726 INFO:teuthology.task.ansible.out: TASK [Check for /usr/bin/python] *********************************************** Tuesday 01 October 2019 02:10:58 +0000 (0:00:00.064) 0:00:00.064 ******* 2019-10-01T02:11:01.149 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.007007", "end": "2019-10-01 02:11:01.115456", "rc": 0, "start": "2019-10-01 02:11:01.108449", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-01T02:11:01.272 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.006201", "end": "2019-10-01 02:11:01.230294", "rc": 0, "start": "2019-10-01 02:11:01.224093", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} TASK [Set ansible_python_interpreter=/usr/bin/python3] ************************* 2019-10-01T02:11:01.273 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:01 +0000 (0:00:02.546) 0:00:02.610 ******* 2019-10-01T02:11:01.301 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:01.322 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:01.332 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-01T02:11:01.349 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:01 +0000 (0:00:00.075) 0:00:02.686 ******* 2019-10-01T02:11:01.361 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:01 +0000 (0:00:00.012) 0:00:02.699 ******* 2019-10-01T02:11:03.273 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-01T02:11:03.297 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.6.124"], "ansible_all_ipv6_addresses": ["fe80::225:90ff:fe32:bf28"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "05/27/2010", "ansible_bios_version": "1.1", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": "ttyS2,115200n81", "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_date_time": {"date": "2019-10-01", "day": "01", "epoch": "1569895863", "hour": "02", "iso8601": "2019-10-01T02:11:03Z", "iso8601_basic": "20191001T021103170239", "iso8601_basic_short": "20191001T021103", "iso8601_micro": "2019-10-01T02:11:03.170522Z", "minute": "11", "month": "10", "second": "03", "time": "02:11:03", "tz": "UTC", "tz_offset": "+0000", "weekday": "Tuesday", "weekday_number": "2", "weeknumber": "39", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.6.124", "alias": "enp4s0", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp4s0", "macaddress": "00:25:90:32:bf:28", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::225:90ff:fe32:bf28", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp4s0", "macaddress": "00:25:90:32:bf:28", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "sda": ["scsi-2001b4d2010960009"], "sda1": ["scsi-2001b4d2010960009-part1"], "sdb": ["scsi-2001b4d2010572303"], "sdc": ["scsi-2001b4d2010865804"], "sdd": ["scsi-2001b4d2010070802"], "sde": ["scsi-2001b4d2070345100"], "sdg": ["scsi-2001b4d2010866207"], "sdh": ["scsi-2001b4d2000000000"]}, "labels": {}, "masters": {"sdf": ["dm-0"], "sdh": ["dm-0"]}, "uuids": {"dm-0": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"], "sda1": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7" 2019-10-01T02:11:03.298 INFO:teuthology.task.ansible.out:], "sdg": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"], "sdh": ["11dd6939-301d-45ff-b781-d2460ec22316"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "labels": [], "masters": [], "uuids": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010960009"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d2010960009-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdb": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010572303"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdc": {"holders": [], "host": "RAID bus controller: Areca Technology C 2019-10-01T02:11:03.300 INFO:teuthology.task.ansible.out:orp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010865804"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdd": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010070802"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sde": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2070345100"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N20GDSAE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdf": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": [], "labels": [], "masters": ["dm-0"], "uuids": []}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": "Seaga 2019-10-01T02:11:03.301 INFO:teuthology.task.ansible.out:te", "virtual": 1}, "sdg": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010866207"], "labels": [], "masters": [], "uuids": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"]}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdh": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2000000000"], "labels": [], "masters": ["dm-0"], "uuids": ["11dd6939-301d-45ff-b781-d2460ec22316"]}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "9VP9NVCL", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "ansible_distribution": "CentOS", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Core", "ansible_distribution_version": "7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_enp4s0": {"active": true, "device": "enp4s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed 2019-10-01T02:11:03.302 INFO:teuthology.task.ansible.out:]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "ipv4": {"address": "172.21.6.124", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::225:90ff:fe32:bf28", "prefix": "64", "scope": "link"}], "macaddress": "00:25:90:32:bf:28", "module": "e1000e", "mtu": 1500, "pciid": "0000:04:00.0", "phc_index": 0, "promisc": false, "speed 2019-10-01T02:11:03.304 INFO:teuthology.task.ansible.out:": 1000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp5s0": {"active": false, "device": "enp5s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", 2019-10-01T02:11:03.306 INFO:teuthology.task.ansible.out: "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "macaddress": "00:25:90:32:bf:29", "module": "e1000e", "mtu": 1500, "pciid": "0000:05:00.0", "phc_index": 1, "promisc": false, "speed": -1, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-vxevihfolhccjtkkykohxjvlqjfsopto ; /usr/bin/python", "SUDO_GID": "1206", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "2", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Sealed-case PC", "ansible_fqdn": "mira063.front.sepia.ceph.com", "ansible_hostname": "mira063", "ansible_hostnqn": "", "ansible_interfaces": ["lo", "enp4s0", "enp5s0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.1.1.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_s 2019-10-01T02:11:03.307 INFO:teuthology.task.ansible.out:egmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb" 2019-10-01T02:11:03.314 INFO:teuthology.task.ansible.out:: {"codename": "Core", "description": "CentOS Linux release 7.6.1810 (Core)", "id": "CentOS", "major_release": "7", "release": "7.6.1810"}, "ansible_lvm": {"lvs": {}, "pvs": {}, "vgs": {}}, "ansible_machine": "x86_64", "ansible_machine_id": "ed19c4d1a216479b9bcb22a6fb70d4b0", "ansible_memfree_mb": 15357, "ansible_memory_mb": {"nocache": {"free": 15619, "used": 256}, "real": {"free": 15357, "total": 15875, "used": 518}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 15875, "ansible_mounts": [{"block_available": 227328295, "block_size": 4096, "block_total": 240324484, "block_used": 12996189, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136696320, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}], "ansible_nodename": "mira063.front.sepia.ceph.com", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": ["ttyS2,115200n81", "ttyS2,115200n81"], "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_na 2019-10-01T02:11:03.316 INFO:teuthology.task.ansible.out:me": "X8SIL", "ansible_product_serial": "0123456789", "ansible_product_uuid": "49434D53-0200-9032-2500-3290250029BF", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi+ufavdTqrQP+qtIN/xYYFLMJ4XZ0MOqh8EYzEncRs1rwFURU5LbFSSkJV5GYz7AaL1PE38lnhmBNFvuhuwTI=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIFx5id/ou+n4zr3+iC1sY6VQ3MgTTDCyHgiaPpAfArxh", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDC0syhJayVLaGcvkxkrH7lkHRQ+9cOZOrjrmRflouJyaEdL6Qo0rCPCL+2XUeLe7iZbSquaSvxF/shLm7hQWBC2wNMPB+BY//Xe0v9kcN7W7IJ7TEkOuQBHx6mWftT5ncTcmsyZtixRos+ejLoQCqJdvNVIAaqHExwvgvFPFRJGloz7ceptcY90+IkonAxuSvtO3vmeEvTb1q5bDZpd4mveI1+XlTLyFhMuLziYImO5UBvStnBGbgRV6BaNQcNvRF59VF25S/nHBQtm3FUGcF5mqdARwJq95F6v7ZP0FVdOyG5JZ0j/XaIXeQysaG3AFNNUCggjNHZKhMxX1+L4KFf", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write 2019-10-01T02:11:03.317 INFO:teuthology.task.ansible.out:", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 68, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-01T02:11:03.396 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:03 +0000 (0:00:02.032) 0:00:04.731 ******* 2019-10-01T02:11:03.432 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.6.120"], "ansible_all_ipv6_addresses": ["fe80::225:90ff:fe09:e2d4"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "05/27/2010", "ansible_bios_version": "1.1", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": "ttyS2,115200n81", "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_date_time": {"date": "2019-10-01", "day": "01", "epoch": "1569895863", "hour": "02", "iso8601": "2019-10-01T02:11:03Z", "iso8601_basic": "20191001T021103328312", "iso8601_basic_short": "20191001T021103", "iso8601_micro": "2019-10-01T02:11:03.328765Z", "minute": "11", "month": "10", "second": "03", "time": "02:11:03", "tz": "UTC", "tz_offset": "+0000", "weekday": "Tuesday", "weekday_number": "2", "weeknumber": "39", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.6.120", "alias": "enp4s0", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp4s0", "macaddress": "00:25:90:09:e2:d4", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::225:90ff:fe09:e2d4", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp4s0", "macaddress": "00:25:90:09:e2:d4", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"sda": ["scsi-2001b4d20114a3400"], "sda1": ["scsi-2001b4d20114a3400-part1"], "sdb": ["scsi-2001b4d20115e3600"], "sdc": ["scsi-2001b4d2001985300"], "sdd": ["scsi-2001b4d2001755c00"], "sde": ["scsi-2001b4d20a0443200"], "sdf": ["scsi-2001b4d2001373100"], "sdg": ["scsi-2001b4d2001475400"], "sdh": ["scsi-2001b4d2001535300"]}, "labels": {}, "masters": {}, "uuids": {"sda1": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"], "sdd": ["08894f69-ea05-45a2-a056-85d846cce4c6"], "sdf": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"], "sdg": [" 2019-10-01T02:11:03.433 INFO:teuthology.task.ansible.out:adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"], "sdh": ["b46ae0b3-5463-4677-893e-f532ede61502"]}}, "ansible_devices": {"sda": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20114a3400"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d20114a3400-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211JT4C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdb": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20115e3600"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211NUVC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdc": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001985300"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2108YSE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, 2019-10-01T02:11:03.435 INFO:teuthology.task.ansible.out: "sdd": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001755c00"], "labels": [], "masters": [], "uuids": ["08894f69-ea05-45a2-a056-85d846cce4c6"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210EWLE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sde": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20a0443200"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZTT2C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdf": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001373100"], "labels": [], "masters": [], "uuids": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N21073AC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdg": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001475400"], "labels": [], "masters": [], "uuids": ["adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0 2019-10-01T02:11:03.436 INFO:teuthology.task.ansible.out:", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210WTDE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdh": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001535300"], "labels": [], "masters": [], "uuids": ["b46ae0b3-5463-4677-893e-f532ede61502"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2103U3E", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "ansible_distribution": "CentOS", "ansible_distribution_file_parsed": true, "ansibl 2019-10-01T02:11:03.438 INFO:teuthology.task.ansible.out:e_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Core", "ansible_distribution_version": "7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_enp4s0": {"active": true, "device": "enp4s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentat 2019-10-01T02:11:03.440 INFO:teuthology.task.ansible.out:ion": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "ipv4": {"address": "172.21.6.120", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::225:90ff:fe09:e2d4", "prefix": "64", "scope": "link"}], "macaddress": "00:25:90:09:e2:d4", "module": "e1000e", "mtu": 1500, "pciid": "0000:04:00.0", "phc_index": 0, "promisc": false, "speed": 1000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp5s0": {"active": false, "device": "enp5s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "of 2019-10-01T02:11:03.444 INFO:teuthology.task.ansible.out:f [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "macaddress": "00:25:90:09:e2:d5", "module": "e1000e", "mtu": 1500, "pciid": "0000:05:00.0", "phc_index": 1, "promisc": false, "speed": -1, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-gfguimkhxgxifyygajasghfslbpoyfou ; /usr/bin/python", "SUDO_GID": "1206", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "2", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Sealed-case PC", "ansible_fqdn": "mira064.front.sepia.ceph.com", "ansible_hostname": "mira064", "ansible_hostnqn": "", "ansible_interfaces": ["lo", "enp4s0", "enp5s0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.1.1.el7.x86_64", "ansible_ 2019-10-01T02:11:03.445 INFO:teuthology.task.ansible.out:lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536 2019-10-01T02:11:03.447 INFO:teuthology.task.ansible.out:, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Core", "description": "CentOS Linux release 7.6.1810 (Core)", "id": "CentOS", "major_release": "7", "release": "7.6.1810"}, "ansible_lvm": {"lvs": {}, "pvs": {}, "vgs": {}}, "ansible_machine": "x86_64", "ansible_machine_id": "ed19c4d1a216479b9bcb22a6fb70d4b0", "ansible_memfree_mb": 15367, "ansible_memory_mb": {"nocache": {"free": 15622, "used": 253}, "real": {"free": 15367, "total": 15875, "used": 508}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 15875, "ansible_mounts": [{"block_available": 227328309, "block_size": 4096, "block_total": 240324484, "block_used": 12996175, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136753664, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}], "ansible_nodename": "mira064.front.sepia.ceph.com", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": ["ttyS2,115200n81", "ttyS2,115200n81"], "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz"], "ansible_processor_cores": 4, 2019-10-01T02:11:03.452 INFO:teuthology.task.ansible.out:"ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "X8SIL", "ansible_product_serial": "0123456789", "ansible_product_uuid": "49434D53-0200-9009-2500-09902500D5E2", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYmlJjcL8RX73FLXiOiL2KeCQYNg+g2eybkFOz/+WA3W6vqwRplE1f3BA2ngC5bq7vIxFJ/DT3ANIw2xkF3b1o=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAILBMDi6qTllO/qTwi1KugokVcL4w9kfeYiLtTfIttc1O", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDE+Ux0+98wzMAe6H0MZQA31ni2vyP0dNZ8EZAFJBqpq0/AITvn7JrRa3WSI6FItIYUdS3tzogVRnnQCjq4/I4CWwxw9pFpYSc0Ikx3r8TVo5qqqrSGkcSRo6DCFZe/GVQZaAAhclynmA/1oPxr62FD7ohW9oWlHOZE/vtcqxKiTjxXTrsxE31qRh8VA3V8x/5/ZijfORMjqQKWKhkJ1oPOOGqi81G7cTatOnA4Ri1PCB8UO34p5MX9ESSUFqaw+BPql5olSoenD3aIEeCCLofuNQ85Gh9nwQSKG+Q08Fb9HSzD3QRBl9JE0j65ZUr8Z6Pf37qv6wg9NNGXTm2Coq69", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boo 2019-10-01T02:11:03.454 INFO:teuthology.task.ansible.out:t", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 61, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": 2019-10-01T02:11:03.455 INFO:teuthology.task.ansible.out:"x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-01T02:11:03.540 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:03 +0000 (0:00:00.145) 0:00:04.877 ******* 2019-10-01T02:11:04.035 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-01T02:11:04.065 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:04 +0000 (0:00:00.525) 0:00:05.403 ******* 2019-10-01T02:11:04.072 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-01T02:11:04.118 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:04 +0000 (0:00:00.052) 0:00:05.455 ******* 2019-10-01T02:11:04.916 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-10-01T02:11:04.918 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-01T02:11:04.957 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:04 +0000 (0:00:00.834) 0:00:06.290 ******* 2019-10-01T02:11:04.971 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-01T02:11:04.975 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:11:05.028 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:05 +0000 (0:00:00.073) 0:00:06.364 ******* 2019-10-01T02:11:05.042 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-10-01T02:11:05.044 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:05.055 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:11:05.092 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:05 +0000 (0:00:00.065) 0:00:06.429 ******* 2019-10-01T02:11:05.099 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:05.149 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:05 +0000 (0:00:00.057) 0:00:06.487 ******* 2019-10-01T02:11:05.732 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-10-01T02:11:05.734 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:11:05.771 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:05 +0000 (0:00:00.621) 0:00:07.108 ******* 2019-10-01T02:11:05.809 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:05 +0000 (0:00:00.037) 0:00:07.146 ******* 2019-10-01T02:11:07.898 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-10-01T02:11:07.898 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569895865.83-270284367158586/source", "state": "file", "uid": 0} 2019-10-01T02:11:07.940 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:07 +0000 (0:00:02.130) 0:00:09.277 ******* 2019-10-01T02:11:07.946 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569895865.79-172675334915092/source", "state": "file", "uid": 0} 2019-10-01T02:11:08.000 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:07 +0000 (0:00:00.059) 0:00:09.337 ******* 2019-10-01T02:11:08.696 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:08.725 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:09.157 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:09.189 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7Tb 2019-10-01T02:11:09.190 INFO:teuthology.task.ansible.out:kfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:09.625 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:09.700 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:10.017 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:10.158 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:10.467 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:10.615 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:10.941 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:11.116 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:11.437 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:11.602 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-01T02:11:11.701 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-10-01T02:11:11.707 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-10-01T02:11:11.882 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:11 +0000 (0:00:03.882) 0:00:13.220 ******* 2019-10-01T02:11:11.911 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:11 +0000 (0:00:00.028) 0:00:13.249 ******* 2019-10-01T02:11:13.083 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-01T02:11:13.091 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.6.124"], "ansible_all_ipv6_addresses": ["fe80::225:90ff:fe32:bf28"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "05/27/2010", "ansible_bios_version": "1.1", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": "ttyS2,115200n81", "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_date_time": {"date": "2019-10-01", "day": "01", "epoch": "1569895872", "hour": "02", "iso8601": "2019-10-01T02:11:12Z", "iso8601_basic": "20191001T021112962663", "iso8601_basic_short": "20191001T021112", "iso8601_micro": "2019-10-01T02:11:12.962939Z", "minute": "11", "month": "10", "second": "12", "time": "02:11:12", "tz": "UTC", "tz_offset": "+0000", "weekday": "Tuesday", "weekday_number": "2", "weeknumber": "39", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.6.124", "alias": "enp4s0", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp4s0", "macaddress": "00:25:90:32:bf:28", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::225:90ff:fe32:bf28", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp4s0", "macaddress": "00:25:90:32:bf:28", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "sda": ["scsi-2001b4d2010960009"], "sda1": ["scsi-2001b4d2010960009-part1"], "sdb": ["scsi-2001b4d2010572303"], "sdc": ["scsi-2001b4d2010865804"], "sdd": ["scsi-2001b4d2010070802"], "sde": ["scsi-2001b4d2070345100"], "sdg": ["scsi-2001b4d2010866207"], "sdh": ["scsi-2001b4d2000000000"]}, "labels": {}, "masters": {"sdf": ["dm-0"], "sdh": ["dm-0"]}, "uuids": {"dm-0": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"], "sda1": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7" 2019-10-01T02:11:13.093 INFO:teuthology.task.ansible.out:], "sdg": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"], "sdh": ["11dd6939-301d-45ff-b781-d2460ec22316"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "labels": [], "masters": [], "uuids": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010960009"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d2010960009-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdb": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010572303"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdc": {"holders": [], "host": "RAID bus controller: Areca Technology C 2019-10-01T02:11:13.094 INFO:teuthology.task.ansible.out:orp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010865804"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdd": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010070802"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sde": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2070345100"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N20GDSAE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdf": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": [], "labels": [], "masters": ["dm-0"], "uuids": []}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": "Seaga 2019-10-01T02:11:13.096 INFO:teuthology.task.ansible.out:te", "virtual": 1}, "sdg": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010866207"], "labels": [], "masters": [], "uuids": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"]}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}, "sdh": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2000000000"], "labels": [], "masters": ["dm-0"], "uuids": ["11dd6939-301d-45ff-b781-d2460ec22316"]}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "9VP9NVCL", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "ansible_distribution": "CentOS", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Core", "ansible_distribution_version": "7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_enp4s0": {"active": true, "device": "enp4s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed 2019-10-01T02:11:13.097 INFO:teuthology.task.ansible.out:]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "ipv4": {"address": "172.21.6.124", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::225:90ff:fe32:bf28", "prefix": "64", "scope": "link"}], "macaddress": "00:25:90:32:bf:28", "module": "e1000e", "mtu": 1500, "pciid": "0000:04:00.0", "phc_index": 0, "promisc": false, "speed 2019-10-01T02:11:13.099 INFO:teuthology.task.ansible.out:": 1000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp5s0": {"active": false, "device": "enp5s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", 2019-10-01T02:11:13.100 INFO:teuthology.task.ansible.out: "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "macaddress": "00:25:90:32:bf:29", "module": "e1000e", "mtu": 1500, "pciid": "0000:05:00.0", "phc_index": 1, "promisc": false, "speed": -1, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-eyzdfjlmmysmbhhmyewfrdnoehuiosil ; /usr/bin/python", "SUDO_GID": "1206", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "2", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Sealed-case PC", "ansible_fqdn": "mira063.front.sepia.ceph.com", "ansible_hostname": "mira063", "ansible_hostnqn": "", "ansible_interfaces": ["lo", "enp4s0", "enp5s0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.1.1.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fi 2019-10-01T02:11:13.102 INFO:teuthology.task.ansible.out:xed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": 2019-10-01T02:11:13.103 INFO:teuthology.task.ansible.out: "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Core", "description": "CentOS Linux release 7.6.1810 (Core)", "id": "CentOS", "major_release": "7", "release": "7.6.1810"}, "ansible_lvm": {"lvs": {}, "pvs": {}, "vgs": {}}, "ansible_machine": "x86_64", "ansible_machine_id": "ed19c4d1a216479b9bcb22a6fb70d4b0", "ansible_memfree_mb": 15350, "ansible_memory_mb": {"nocache": {"free": 15615, "used": 260}, "real": {"free": 15350, "total": 15875, "used": 525}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 15875, "ansible_mounts": [{"block_available": 227328280, "block_size": 4096, "block_total": 240324484, "block_used": 12996204, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136634880, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}], "ansible_nodename": "mira063.front.sepia.ceph.com", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": ["ttyS2,115200n81", "ttyS2,115200n81"], "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, " 2019-10-01T02:11:13.105 INFO:teuthology.task.ansible.out:ansible_processor_vcpus": 8, "ansible_product_name": "X8SIL", "ansible_product_serial": "0123456789", "ansible_product_uuid": "49434D53-0200-9032-2500-3290250029BF", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi+ufavdTqrQP+qtIN/xYYFLMJ4XZ0MOqh8EYzEncRs1rwFURU5LbFSSkJV5GYz7AaL1PE38lnhmBNFvuhuwTI=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIFx5id/ou+n4zr3+iC1sY6VQ3MgTTDCyHgiaPpAfArxh", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDC0syhJayVLaGcvkxkrH7lkHRQ+9cOZOrjrmRflouJyaEdL6Qo0rCPCL+2XUeLe7iZbSquaSvxF/shLm7hQWBC2wNMPB+BY//Xe0v9kcN7W7IJ7TEkOuQBHx6mWftT5ncTcmsyZtixRos+ejLoQCqJdvNVIAaqHExwvgvFPFRJGloz7ceptcY90+IkonAxuSvtO3vmeEvTb1q5bDZpd4mveI1+XlTLyFhMuLziYImO5UBvStnBGbgRV6BaNQcNvRF59VF25S/nHBQtm3FUGcF5mqdARwJq95F6v7ZP0FVdOyG5JZ0j/XaIXeQysaG3AFNNUCggjNHZKhMxX1+L4KFf", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_con 2019-10-01T02:11:13.106 INFO:teuthology.task.ansible.out:fig", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 78, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-01T02:11:13.185 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:01.267) 0:00:14.516 ******* 2019-10-01T02:11:13.207 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-01T02:11:13.208 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.251 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.072) 0:00:14.589 ******* 2019-10-01T02:11:13.262 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-01T02:11:13.269 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.6.120"], "ansible_all_ipv6_addresses": ["fe80::225:90ff:fe09:e2d4"], "ansible_apparmor": {"status": "disabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "05/27/2010", "ansible_bios_version": "1.1", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": "ttyS2,115200n81", "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_date_time": {"date": "2019-10-01", "day": "01", "epoch": "1569895873", "hour": "02", "iso8601": "2019-10-01T02:11:13Z", "iso8601_basic": "20191001T021113131151", "iso8601_basic_short": "20191001T021113", "iso8601_micro": "2019-10-01T02:11:13.131426Z", "minute": "11", "month": "10", "second": "13", "time": "02:11:13", "tz": "UTC", "tz_offset": "+0000", "weekday": "Tuesday", "weekday_number": "2", "weeknumber": "39", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.6.120", "alias": "enp4s0", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp4s0", "macaddress": "00:25:90:09:e2:d4", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::225:90ff:fe09:e2d4", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp4s0", "macaddress": "00:25:90:09:e2:d4", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"sda": ["scsi-2001b4d20114a3400"], "sda1": ["scsi-2001b4d20114a3400-part1"], "sdb": ["scsi-2001b4d20115e3600"], "sdc": ["scsi-2001b4d2001985300"], "sdd": ["scsi-2001b4d2001755c00"], "sde": ["scsi-2001b4d20a0443200"], "sdf": ["scsi-2001b4d2001373100"], "sdg": ["scsi-2001b4d2001475400"], "sdh": ["scsi-2001b4d2001535300"]}, "labels": {}, "masters": {}, "uuids": {"sda1": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"], "sdd": ["08894f69-ea05-45a2-a056-85d846cce4c6"], "sdf": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"], "sdg": [" 2019-10-01T02:11:13.270 INFO:teuthology.task.ansible.out:adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"], "sdh": ["b46ae0b3-5463-4677-893e-f532ede61502"]}}, "ansible_devices": {"sda": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20114a3400"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d20114a3400-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211JT4C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdb": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20115e3600"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211NUVC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdc": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001985300"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2108YSE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, 2019-10-01T02:11:13.272 INFO:teuthology.task.ansible.out: "sdd": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001755c00"], "labels": [], "masters": [], "uuids": ["08894f69-ea05-45a2-a056-85d846cce4c6"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210EWLE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sde": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20a0443200"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZTT2C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdf": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001373100"], "labels": [], "masters": [], "uuids": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N21073AC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdg": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001475400"], "labels": [], "masters": [], "uuids": ["adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0 2019-10-01T02:11:13.273 INFO:teuthology.task.ansible.out:", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210WTDE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}, "sdh": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001535300"], "labels": [], "masters": [], "uuids": ["b46ae0b3-5463-4677-893e-f532ede61502"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2103U3E", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "ansible_distribution": "CentOS", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/redhat-release", "ansible_distribution_file_variety": "RedHat", "ansible_distribution_major_version": "7", "ansible_distribution_release": "Core", "ansible_distribution_version": "7", "ansible_dns": {"nameservers": ["172.21.0.1", "172.21.0.2"], "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_enp4s0": {"active": true, "device": "enp4s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter 2019-10-01T02:11:13.275 INFO:teuthology.task.ansible.out:": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "ipv4": {"address": "172.21.6.120", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::225:90ff:fe09:e2d4", "prefix": "64", "scope": "link"}], "macaddress": "00:25:90:09:e2:d4", "module": "e1000e", "mtu": 1500, "pciid": "0000:04:00.0", "phc_index": 0, "promisc": false, "speed": 1000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp5s0": {"active": false, "device": "enp5s0", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", 2019-10-01T02:11:13.276 INFO:teuthology.task.ansible.out:"highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_l4_sync", "ptp_v2_l4_delay_req", "ptp_v2_l2_sync", "ptp_v2_l2_delay_req", "ptp_v2_event", "ptp_v2_sync", "ptp_v2_delay_req"], "macaddress": "00:25:90:09:e2:d5", "module": "e1000e", "mtu": 1500, "pciid": "0000:05:00.0", "phc_index": 1, "promisc": false, "speed": -1, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_softw 2019-10-01T02:11:13.278 INFO:teuthology.task.ansible.out:are", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US.UTF-8", "LOGNAME": "root", "MAIL": "/var/mail/ubuntu", "PATH": "/sbin:/bin:/usr/sbin:/usr/bin", "PWD": "/home/ubuntu", "QTDIR": "/usr/lib64/qt-3.3", "SHELL": "/bin/bash", "SHLVL": "1", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-cghfnufnkuqadfnicteclxexnholroyv ; /usr/bin/python", "SUDO_GID": "1206", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root", "XDG_SESSION_ID": "2", "_": "/usr/bin/python"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Sealed-case PC", "ansible_fqdn": "mira064.front.sepia.ceph.com", "ansible_hostname": "mira064", "ansible_hostnqn": "", "ansible_interfaces": ["lo", "enp4s0", "enp5s0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "", "ansible_kernel": "3.10.0-1062.1.1.el7.x86_64", "ansible_lo": {"active": true, "device": "lo", "features": {"busy_poll": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "rece 2019-10-01T02:11:13.279 INFO:teuthology.task.ansible.out:ive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_gro_hw": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_sit_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "on", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "Core", "description": "CentOS Linux release 7.6.1810 (Core)", "id": "CentOS", "major_release": "7", "release": "7.6.1810"}, "ansible_lvm": {"lvs": {}, "pvs": {}, "vgs": {}}, "ansible_machine": "x86_64", "ansible_machine_id": "ed19c4d1a216479b9bcb22a6fb70d4b0", "an 2019-10-01T02:11:13.281 INFO:teuthology.task.ansible.out:sible_memfree_mb": 15362, "ansible_memory_mb": {"nocache": {"free": 15620, "used": 255}, "real": {"free": 15362, "total": 15875, "used": 513}, "swap": {"cached": 0, "free": 0, "total": 0, "used": 0}}, "ansible_memtotal_mb": 15875, "ansible_mounts": [{"block_available": 227328295, "block_size": 4096, "block_total": 240324484, "block_used": 12996189, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136696320, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}], "ansible_nodename": "mira064.front.sepia.ceph.com", "ansible_os_family": "RedHat", "ansible_pkg_mgr": "yum", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1062.1.1.el7.x86_64", "console": ["ttyS2,115200n81", "ttyS2,115200n81"], "crashkernel": "auto", "ro": true, "root": "UUID=254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU X3440 @ 2.53GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "X8SIL", "ansible_product_serial": "0123456789", "ansible_product_uuid": "49434D53-0200-9009-2500-09902500D5E2", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "m 2019-10-01T02:11:13.282 INFO:teuthology.task.ansible.out:icro": 5, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 5, "final", 0]}, "ansible_python_version": "2.7.5", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"config_mode": "permissive", "mode": "permissive", "policyvers": 31, "status": "enabled", "type": "targeted"}, "ansible_selinux_python_present": true, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYmlJjcL8RX73FLXiOiL2KeCQYNg+g2eybkFOz/+WA3W6vqwRplE1f3BA2ngC5bq7vIxFJ/DT3ANIw2xkF3b1o=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAILBMDi6qTllO/qTwi1KugokVcL4w9kfeYiLtTfIttc1O", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDE+Ux0+98wzMAe6H0MZQA31ni2vyP0dNZ8EZAFJBqpq0/AITvn7JrRa3WSI6FItIYUdS3tzogVRnnQCjq4/I4CWwxw9pFpYSc0Ikx3r8TVo5qqqrSGkcSRo6DCFZe/GVQZaAAhclynmA/1oPxr62FD7ohW9oWlHOZE/vtcqxKiTjxXTrsxE31qRh8VA3V8x/5/ZijfORMjqQKWKhkJ1oPOOGqi81G7cTatOnA4Ri1PCB8UO34p5MX9ESSUFqaw+BPql5olSoenD3aIEeCCLofuNQ85Gh9nwQSKG+Q08Fb9HSzD3QRBl9JE0j65ZUr8Z6Pf37qv6wg9NNGXTm2Coq69", "ansible_swapfree_mb": 0, "ansible_swaptotal_mb": 0, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "35", "36+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 70, "ansible_user_dir": "/root", "ansible_user_gecos": "root" 2019-10-01T02:11:13.284 INFO:teuthology.task.ansible.out:, "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-01T02:11:13.288 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-01T02:11:13.289 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.343 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.090) 0:00:14.679 ******* 2019-10-01T02:11:13.386 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-01T02:11:13.387 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.423 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.081) 0:00:14.761 ******* 2019-10-01T02:11:13.471 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.047) 0:00:14.808 ******* 2019-10-01T02:11:13.492 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.528 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.056) 0:00:14.865 ******* 2019-10-01T02:11:13.534 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-01T02:11:13.535 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.569 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.041) 0:00:14.907 ******* 2019-10-01T02:11:13.611 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-01T02:11:13.613 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.656 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.084) 0:00:14.991 ******* 2019-10-01T02:11:13.684 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-01T02:11:13.686 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.736 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.082) 0:00:15.073 ******* 2019-10-01T02:11:13.766 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:11:13.812 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:13 +0000 (0:00:00.075) 0:00:15.149 ******* 2019-10-01T02:11:13.991 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-10-01T02:11:13.993 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:11:14.031 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:14 +0000 (0:00:00.219) 0:00:15.368 ******* 2019-10-01T02:11:14.296 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:11:14.335 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:11:14 +0000 (0:00:00.304) 0:00:15.673 ******* 2019-10-01T02:11:14.540 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-01T02:11:14.541 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-01T02:11:14.828 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-01T02:11:14.958 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-01T02:11:15.321 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-01T02:11:15.419 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-01T02:11:15.803 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-01T02:11:15.868 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-01T02:11:16.297 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-01T02:11:16.354 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-01T02:11:16.800 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-01T02:11:16.853 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-01T02:11:17.289 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-01T02:11:17.410 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-01T02:11:17.787 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-01T02:11:17.959 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "name": "akraitma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-01T02:11:18.371 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "name": "akraitma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-01T02:11:18.470 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-01T02:11:18.833 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-01T02:11:18.978 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-01T02:11:19.330 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-01T02:11:19.440 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-01T02:11:19.813 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-01T02:11:19.944 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-01T02:11:20.281 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-01T02:11:20.447 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-01T02:11:20.741 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-01T02:11:20.937 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-01T02:11:21.262 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-01T02:11:21.457 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-01T02:11:21.749 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-01T02:11:21.962 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-01T02:11:22.270 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-01T02:11:22.481 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-01T02:11:22.781 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-01T02:11:22.998 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-01T02:11:23.263 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-01T02:11:23.483 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-01T02:11:23.786 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-01T02:11:23.982 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-01T02:11:24.288 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-01T02:11:24.466 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-01T02:11:24.790 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-01T02:11:24.973 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-01T02:11:25.274 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-01T02:11:25.468 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-01T02:11:25.758 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-01T02:11:25.938 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-01T02:11:26.266 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-01T02:11:26.396 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-01T02:11:26.764 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-01T02:11:26.819 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-01T02:11:27.211 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-01T02:11:27.214 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-01T02:11:27.639 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-01T02:11:27.736 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-01T02:11:28.143 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-01T02:11:28.244 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-01T02:11:28.620 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-01T02:11:28.733 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-01T02:11:29.157 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-01T02:11:29.228 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-01T02:11:29.642 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-01T02:11:29.739 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-01T02:11:30.101 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-01T02:11:30.225 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-01T02:11:30.546 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-01T02:11:30.704 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-01T02:11:30.966 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-01T02:11:31.181 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-01T02:11:31.469 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-01T02:11:31.652 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-01T02:11:31.976 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-01T02:11:32.152 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-01T02:11:32.460 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-01T02:11:32.635 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-01T02:11:32.974 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-01T02:11:33.128 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-01T02:11:33.402 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-01T02:11:33.586 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-01T02:11:33.808 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-01T02:11:34.063 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-01T02:11:34.231 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-01T02:11:34.547 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-01T02:11:34.632 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-01T02:11:35.038 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-01T02:11:35.102 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-01T02:11:35.556 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1143} ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-01T02:11:35.977 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-01T02:11:36.056 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-01T02:11:36.428 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-01T02:11:36.545 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-01T02:11:36.915 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-01T02:11:36.999 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-01T02:11:37.427 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-01T02:11:37.500 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-01T02:11:37.907 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-01T02:11:37.983 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-01T02:11:38.413 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-01T02:11:38.484 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-01T02:11:38.881 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-01T02:11:38.957 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-01T02:11:39.367 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-01T02:11:39.454 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-01T02:11:39.834 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-01T02:11:39.959 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-01T02:11:40.283 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-01T02:11:40.457 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-01T02:11:40.764 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-01T02:11:40.972 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-01T02:11:41.261 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-01T02:11:41.486 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-01T02:11:41.761 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-01T02:11:41.988 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-01T02:11:42.283 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-01T02:11:42.481 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-01T02:11:42.735 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-01T02:11:42.979 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-01T02:11:43.245 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-01T02:11:43.503 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-01T02:11:43.750 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-01T02:11:44.027 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-01T02:11:44.225 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-01T02:11:44.500 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-01T02:11:44.724 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-01T02:11:44.988 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-01T02:11:45.168 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-01T02:11:45.477 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-01T02:11:45.663 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-01T02:11:45.977 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-01T02:11:46.162 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-01T02:11:46.464 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-01T02:11:46.682 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-01T02:11:46.960 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-01T02:11:47.211 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-01T02:11:47.459 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-01T02:11:47.731 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-01T02:11:47.964 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-01T02:11:48.221 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-01T02:11:48.444 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-01T02:11:48.727 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-01T02:11:48.941 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-01T02:11:49.205 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-01T02:11:49.441 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-01T02:11:49.696 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-01T02:11:49.946 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-01T02:11:50.225 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-01T02:11:50.463 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-01T02:11:50.735 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-01T02:11:50.942 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-01T02:11:51.261 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-01T02:11:51.419 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-01T02:11:51.771 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-01T02:11:51.919 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-01T02:11:52.277 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-01T02:11:52.387 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-01T02:11:52.762 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-01T02:11:52.881 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-01T02:11:53.271 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-01T02:11:53.358 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-01T02:11:53.777 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-01T02:11:53.811 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-01T02:11:54.242 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-01T02:11:54.275 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-01T02:11:54.718 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-01T02:11:54.729 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-01T02:11:55.222 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-01T02:11:55.223 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-01T02:11:55.728 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-01T02:11:55.740 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-01T02:11:56.224 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-01T02:11:56.257 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-01T02:11:56.729 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-01T02:11:56.765 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-01T02:11:57.214 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-01T02:11:57.247 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-01T02:11:57.664 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-01T02:11:57.697 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-01T02:11:58.037 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-01T02:11:58.164 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-01T02:11:58.510 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-01T02:11:58.673 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-01T02:11:58.999 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-01T02:11:59.172 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-01T02:11:59.491 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-01T02:11:59.668 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-01T02:11:59.982 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-01T02:12:00.181 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-01T02:12:00.478 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-01T02:12:00.623 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-01T02:12:00.959 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-01T02:12:01.071 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-01T02:12:01.412 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-01T02:12:01.547 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-01T02:12:01.879 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-01T02:12:02.016 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-01T02:12:02.362 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-01T02:12:02.510 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-01T02:12:02.843 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-01T02:12:03.003 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-01T02:12:03.331 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-01T02:12:03.466 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-01T02:12:03.830 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-01T02:12:03.944 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-01T02:12:04.327 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-01T02:12:04.453 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-01T02:12:04.800 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-01T02:12:04.950 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-01T02:12:05.260 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-01T02:12:05.450 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-01T02:12:05.725 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-01T02:12:05.956 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-01T02:12:06.158 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-01T02:12:06.475 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-01T02:12:06.609 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "name": "lmb", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-01T02:12:06.974 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "name": "lmb", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-01T02:12:07.071 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-01T02:12:07.464 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-01T02:12:07.571 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-01T02:12:08.030 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-01T02:12:08.055 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-01T02:12:08.535 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-01T02:12:08.556 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-01T02:12:09.024 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2019-10-01T02:12:09.142 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-01T02:12:09.175 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:54.840) 0:01:10.513 ******* 2019-10-01T02:12:09.210 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-01T02:12:09.248 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:00.072) 0:01:10.586 ******* 2019-10-01T02:12:09.311 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-01T02:12:09.318 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "troc 2019-10-01T02:12:09.320 INFO:teuthology.task.ansible.out:iny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04ok 2019-10-01T02:12:09.321 INFO:teuthology.task.ansible.out:Mj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd 2019-10-01T02:12:09.323 INFO:teuthology.task.ansible.out:", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowi 2019-10-01T02:12:09.324 INFO:teuthology.task.ansible.out:lkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2 2019-10-01T02:12:09.326 INFO:teuthology.task.ansible.out:caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d4 2019-10-01T02:12:09.327 INFO:teuthology.task.ansible.out:33d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, 2019-10-01T02:12:09.329 INFO:teuthology.task.ansible.out:{"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hche 2019-10-01T02:12:09.330 INFO:teuthology.task.ansible.out:n", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laur 2019-10-01T02:12:09.332 INFO:teuthology.task.ansible.out:a", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983 2019-10-01T02:12:09.333 INFO:teuthology.task.ansible.out:d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4 2019-10-01T02:12:09.334 INFO:teuthology.task.ansible.out:b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}]}, "changed": false} 2019-10-01T02:12:09.370 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-10-01T02:12:09.372 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:00.122) 0:01:10.708 ******* 2019-10-01T02:12:09.638 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2019-10-01T02:12:09.792 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:00.421) 0:01:11.130 ******* 2019-10-01T02:12:09.821 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-01T02:12:09.869 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:00.076) 0:01:11.206 ******* 2019-10-01T02:12:09.928 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-01T02:12:09.934 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "troc 2019-10-01T02:12:09.935 INFO:teuthology.task.ansible.out:iny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04ok 2019-10-01T02:12:09.937 INFO:teuthology.task.ansible.out:Mj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd 2019-10-01T02:12:09.938 INFO:teuthology.task.ansible.out:", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowi 2019-10-01T02:12:09.940 INFO:teuthology.task.ansible.out:lkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2 2019-10-01T02:12:09.941 INFO:teuthology.task.ansible.out:caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {" 2019-10-01T02:12:09.942 INFO:teuthology.task.ansible.out:name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbaz 2019-10-01T02:12:09.944 INFO:teuthology.task.ansible.out:j@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253 2019-10-01T02:12:09.945 INFO:teuthology.task.ansible.out:574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg 2019-10-01T02:12:09.947 INFO:teuthology.task.ansible.out:db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEX 2019-10-01T02:12:09.948 INFO:teuthology.task.ansible.out:La7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsa 2019-10-01T02:12:09.950 INFO:teuthology.task.ansible.out:lomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}]}, "changed": false} 2019-10-01T02:12:09.988 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:09 +0000 (0:00:00.118) 0:01:11.325 ******* 2019-10-01T02:12:10.411 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:12:10.428 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [mira063.front.sepia.ceph.com] => {"after": "601a27ecf2f12d023d7071016a164b1c9fdecf3c", "attempts": 1, "before": "601a27ecf2f12d023d7071016a164b1c9fdecf3c", "changed": false, "remote_url_changed": false} 2019-10-01T02:12:10.489 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:10 +0000 (0:00:00.500) 0:01:11.826 ******* 2019-10-01T02:12:10.722 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:12:10.735 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"after": "601a27ecf2f12d023d7071016a164b1c9fdecf3c", "attempts": 1, "before": "601a27ecf2f12d023d7071016a164b1c9fdecf3c", "changed": false, "remote_url_changed": false} 2019-10-01T02:12:10.777 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:12:10 +0000 (0:00:00.288) 0:01:12.114 ******* 2019-10-01T02:12:11.009 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-01T02:12:11.010 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-01T02:12:11.332 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-01T02:12:11.442 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-01T02:12:11.832 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-01T02:12:11.885 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-01T02:12:12.326 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-01T02:12:12.349 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-01T02:12:12.842 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "vali 2019-10-01T02:12:12.843 INFO:teuthology.task.ansible.out:date_certs": true} 2019-10-01T02:12:12.875 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-01T02:12:13.318 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-01T02:12:13.373 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "vali 2019-10-01T02:12:13.374 INFO:teuthology.task.ansible.out:date_certs": true} 2019-10-01T02:12:13.815 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEd 2019-10-01T02:12:13.817 INFO:teuthology.task.ansible.out:z+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-01T02:12:13.859 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-01T02:12:14.290 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-01T02:12:14.380 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEd 2019-10-01T02:12:14.382 INFO:teuthology.task.ansible.out:z+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-01T02:12:14.735 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-01T02:12:14.747 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:14.883 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-01T02:12:15.227 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfM 2019-10-01T02:12:15.229 INFO:teuthology.task.ansible.out:orbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-01T02:12:15.239 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.250 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.251 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.262 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.411 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-01T02:12:15.423 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.751 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-01T02:12:15.861 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfM 2019-10-01T02:12:15.862 INFO:teuthology.task.ansible.out:orbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.873 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.894 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:15.895 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:12:16.242 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2v 2019-10-01T02:12:16.244 INFO:teuthology.task.ansible.out:tSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-01T02:12:16.377 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-01T02:12:16.740 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-01T02:12:16.920 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2v 2019-10-01T02:12:16.924 INFO:teuthology.task.ansible.out:tSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-01T02:12:17.248 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-01T02:12:17.365 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-01T02:12:17.726 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-01T02:12:17.855 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-01T02:12:18.228 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-01T02:12:18.358 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-01T02:12:18.742 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-01T02:12:18.808 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-01T02:12:19.256 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-01T02:12:19.258 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-01T02:12:19.689 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-01T02:12:19.784 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2019-10-01T02:12:19.786 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-01T02:12:20.090 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2019-10-01T02:12:20.091 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-01T02:12:20.268 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRho 2019-10-01T02:12:20.270 INFO:teuthology.task.ansible.out:BYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-01T02:12:20.551 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRho 2019-10-01T02:12:20.553 INFO:teuthology.task.ansible.out:BYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-01T02:12:20.774 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-01T02:12:20.984 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-01T02:12:21.304 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-01T02:12:21.412 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-01T02:12:21.787 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih 2019-10-01T02:12:21.789 INFO:teuthology.task.ansible.out:46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-01T02:12:21.835 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih 2019-10-01T02:12:21.837 INFO:teuthology.task.ansible.out:46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-01T02:12:22.276 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-01T02:12:22.326 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-01T02:12:22.787 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-01T02:12:22.819 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-01T02:12:23.256 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": 2019-10-01T02:12:23.258 INFO:teuthology.task.ansible.out:"present", "user": "xiaoxichen", "validate_certs": true} 2019-10-01T02:12:23.688 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-01T02:12:23.742 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-01T02:12:24.098 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-01T02:12:24.203 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-01T02:12:24.518 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2Cf 2019-10-01T02:12:24.520 INFO:teuthology.task.ansible.out:QvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-01T02:12:24.699 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2Cf 2019-10-01T02:12:24.701 INFO:teuthology.task.ansible.out:QvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-01T02:12:24.995 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+oD 2019-10-01T02:12:24.996 INFO:teuthology.task.ansible.out:lH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-01T02:12:25.194 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+oD 2019-10-01T02:12:25.196 INFO:teuthology.task.ansible.out:lH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-01T02:12:25.425 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-01T02:12:25.689 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-01T02:12:25.836 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-01T02:12:26.169 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-01T02:12:26.254 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-01T02:12:26.660 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-01T02:12:26.713 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-01T02:12:27.123 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-01T02:12:27.124 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-01T02:12:27.594 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-01T02:12:27.612 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-01T02:12:28.039 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-01T02:12:28.134 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-01T02:12:28.465 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-01T02:12:28.644 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-01T02:12:28.885 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-01T02:12:29.177 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-01T02:12:29.303 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-01T02:12:29.639 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-01T02:12:29.748 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-01T02:12:30.131 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AK 2019-10-01T02:12:30.132 INFO:teuthology.task.ansible.out:Y1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-01T02:12:30.184 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AK 2019-10-01T02:12:30.186 INFO:teuthology.task.ansible.out:Y1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-01T02:12:30.622 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZ 2019-10-01T02:12:30.623 INFO:teuthology.task.ansible.out:yKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-01T02:12:30.666 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZ 2019-10-01T02:12:30.668 INFO:teuthology.task.ansible.out:yKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-01T02:12:31.080 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-01T02:12:31.105 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-01T02:12:31.557 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-01T02:12:31.558 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-01T02:12:31.951 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-01T02:12:31.994 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-01T02:12:32.406 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-01T02:12:32.448 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-01T02:12:32.827 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-01T02:12:32.870 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-01T02:12:33.247 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-01T02:12:33.312 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-01T02:12:33.722 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-01T02:12:33.795 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-01T02:12:34.182 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-01T02:12:34.288 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-01T02:12:34.664 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw 2019-10-01T02:12:34.666 INFO:teuthology.task.ansible.out:+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMm 2019-10-01T02:12:34.667 INFO:teuthology.task.ansible.out:bBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-01T02:12:34.764 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw 2019-10-01T02:12:34.766 INFO:teuthology.task.ansible.out:+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMm 2019-10-01T02:12:34.767 INFO:teuthology.task.ansible.out:bBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-01T02:12:35.158 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-01T02:12:35.201 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-01T02:12:35.585 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-01T02:12:35.628 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-01T02:12:36.067 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-01T02:12:36.069 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-01T02:12:36.523 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys", 2019-10-01T02:12:36.525 INFO:teuthology.task.ansible.out: "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-01T02:12:36.546 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys", 2019-10-01T02:12:36.547 INFO:teuthology.task.ansible.out: "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-01T02:12:36.934 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-01T02:12:37.034 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-01T02:12:37.367 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-01T02:12:37.516 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-01T02:12:37.842 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-01T02:12:37.969 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-01T02:12:38.270 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-01T02:12:38.388 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-01T02:12:38.683 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-01T02:12:38.889 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-01T02:12:39.091 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-01T02:12:39.424 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-01T02:12:39.560 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": 2019-10-01T02:12:39.561 INFO:teuthology.task.ansible.out: "gabrioux", "validate_certs": true} 2019-10-01T02:12:39.939 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": 2019-10-01T02:12:39.940 INFO:teuthology.task.ansible.out: "gabrioux", "validate_certs": true} 2019-10-01T02:12:40.064 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-01T02:12:40.416 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-01T02:12:40.565 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-01T02:12:40.898 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-01T02:12:41.086 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33 2019-10-01T02:12:41.087 INFO:teuthology.task.ansible.out:Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-01T02:12:41.396 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33 2019-10-01T02:12:41.397 INFO:teuthology.task.ansible.out:Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-01T02:12:41.541 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-01T02:12:41.897 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-01T02:12:41.982 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-01T02:12:42.393 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-01T02:12:42.405 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-01T02:12:42.876 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-01T02:12:42.877 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-01T02:12:43.349 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-01T02:12:43.354 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-01T02:12:43.791 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-01T02:12:43.849 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-01T02:12:44.244 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-01T02:12:44.350 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-01T02:12:44.685 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-01T02:12:44.847 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-01T02:12:45.145 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-01T02:12:45.352 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-01T02:12:45.645 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-01T02:12:45.822 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-01T02:12:46.156 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-01T02:12:46.313 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-01T02:12:46.654 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-01T02:12:46.821 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-01T02:12:47.115 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-01T02:12:47.285 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-01T02:12:47.574 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-01T02:12:47.736 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-01T02:12:48.071 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-01T02:12:48.202 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-01T02:12:48.526 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-01T02:12:48.630 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-01T02:12:48.979 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-01T02:12:49.106 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-01T02:12:49.467 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-01T02:12:49.592 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-01T02:12:49.965 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-01T02:12:50.101 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-01T02:12:50.453 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-01T02:12:50.557 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-01T02:12:50.993 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-01T02:12:51.023 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-01T02:12:51.462 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-01T02:12:51.464 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-01T02:12:51.910 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-01T02:12:51.964 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-01T02:12:52.318 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-01T02:12:52.455 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-01T02:12:52.808 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-01T02:12:52.932 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-01T02:12:53.294 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-01T02:12:53.421 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-01T02:12:53.758 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-01T02:12:53.905 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-01T02:12:54.255 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-01T02:12:54.428 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-01T02:12:54.743 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-01T02:12:54.954 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-01T02:12:55.222 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-01T02:12:55.464 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost. 2019-10-01T02:12:55.466 INFO:teuthology.task.ansible.out:localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-01T02:12:55.739 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost. 2019-10-01T02:12:55.741 INFO:teuthology.task.ansible.out:localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-01T02:12:56.002 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsm 2019-10-01T02:12:56.004 INFO:teuthology.task.ansible.out:Xq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-01T02:12:56.272 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsm 2019-10-01T02:12:56.274 INFO:teuthology.task.ansible.out:Xq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-01T02:12:56.476 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-01T02:12:56.736 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-01T02:12:57.010 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-01T02:12:57.227 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-01T02:12:57.498 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-01T02:12:57.707 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-01T02:12:58.023 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-01T02:12:58.180 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-01T02:12:58.515 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-01T02:12:58.650 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-01T02:12:59.003 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-01T02:12:59.077 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-01T02:12:59.484 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-01T02:12:59.581 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-01T02:12:59.980 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-01T02:13:00.074 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-01T02:13:00.465 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-01T02:13:00.534 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-01T02:13:00.966 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-01T02:13:01.044 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-01T02:13:01.482 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-01T02:13:01.547 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-01T02:13:01.981 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-01T02:13:02.076 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-01T02:13:02.157 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:02 +0000 (0:00:51.378) 0:02:03.492 ******* 2019-10-01T02:13:02.186 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:02 +0000 (0:00:00.031) 0:02:03.524 ******* 2019-10-01T02:13:02.222 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-01T02:13:02.224 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.227 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.238 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.239 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.241 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.251 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.263 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.277 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.279 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d24945 2019-10-01T02:13:02.280 INFO:teuthology.task.ansible.out:0ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.291 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.291 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.293 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.306 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.766 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-01T02:13:02.778 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:02.826 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-01T02:13:02.839 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:03.255 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-01T02:13:03.298 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-01T02:13:03.744 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-01T02:13:03.757 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-01T02:13:04.182 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-01T02:13:04.194 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-01T02:13:04.662 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-01T02:13:04.663 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.674 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.686 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-01T02:13:04.686 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.688 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.698 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.699 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.700 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.711 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.711 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.713 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.723 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.724 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.725 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.736 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.737 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.738 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.749 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.750 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.751 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.762 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.763 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.777 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.777 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.779 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.781 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.792 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.794 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.795 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.808 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.809 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.811 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\n"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.812 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.825 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.826 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.827 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.829 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.840 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.841 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.842 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.854 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.855 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.856 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.867 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.868 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.870 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.882 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.883 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.884 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.898 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.900 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d9 2019-10-01T02:13:04.901 INFO:teuthology.task.ansible.out:5ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.911 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.911 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.922 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.923 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.925 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.926 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.944 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.951 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.951 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.953 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.954 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.956 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.966 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.967 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.968 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.971 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.981 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.981 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:04.996 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.000 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.001 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.002 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.004 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.017 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.018 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.019 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.022 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.023 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.034 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.035 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.037 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.038 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.048 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.049 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.051 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.061 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.062 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.063 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.074 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.075 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.089 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.090 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.091 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.093 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.103 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.103 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.105 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.106 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.117 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.129 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.153 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2019-10-01T02:13:05.155 INFO:teuthology.task.ansible.out:e": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.169 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.170 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.172 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.182 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.183 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.184 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.195 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.196 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.208 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.209 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.210 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.214 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.215 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.216 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.228 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.228 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.230 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.241 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.242 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.245 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.256 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.258 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.268 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.269 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.270 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.272 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.288 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.se 2019-10-01T02:13:05.290 INFO:teuthology.task.ansible.out:pia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.301 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.303 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.304 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.306 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.307 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.318 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.321 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.329 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.330 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.332 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.344 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.345 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.360 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.371 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.372 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.375 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.376 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.379 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.401 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.402 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.457 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:05 +0000 (0:00:03.270) 0:02:06.794 ******* 2019-10-01T02:13:05.490 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:05 +0000 (0:00:00.033) 0:02:06.828 ******* 2019-10-01T02:13:05.498 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.543 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:05 +0000 (0:00:00.045) 0:02:06.873 ******* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:05.604 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:05 +0000 (0:00:00.061) 0:02:06.935 ******* 2019-10-01T02:13:06.016 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-01T02:13:06.018 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-01T02:13:06.132 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-01T02:13:06.390 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-01T02:13:06.645 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-01T02:13:06.869 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-01T02:13:07.141 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-01T02:13:07.365 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-01T02:13:07.640 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-01T02:13:07.852 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-01T02:13:08.119 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-01T02:13:08.271 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-01T02:13:08.636 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-01T02:13:08.688 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-01T02:13:09.099 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-01T02:13:09.144 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-01T02:13:09.513 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-01T02:13:09.610 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-01T02:13:09.901 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-01T02:13:10.092 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-01T02:13:10.376 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-01T02:13:10.581 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-01T02:13:10.874 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-01T02:13:11.072 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-01T02:13:11.350 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-01T02:13:11.556 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-01T02:13:11.747 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-01T02:13:12.038 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-01T02:13:12.145 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-01T02:13:12.523 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-01T02:13:12.567 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-01T02:13:12.956 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-01T02:13:13.000 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-01T02:13:13.367 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-01T02:13:13.519 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-01T02:13:13.762 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-01T02:13:13.996 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-01T02:13:14.240 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-01T02:13:14.440 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-01T02:13:14.741 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-01T02:13:14.856 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-01T02:13:15.236 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-01T02:13:15.302 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-01T02:13:15.726 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-01T02:13:15.792 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-01T02:13:16.139 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-01T02:13:16.277 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-01T02:13:16.548 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-01T02:13:16.760 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-01T02:13:16.974 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-01T02:13:17.274 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-01T02:13:17.467 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-01T02:13:17.752 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-01T02:13:17.915 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-01T02:13:17.989 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:17 +0000 (0:00:12.391) 0:02:19.326 ******* 2019-10-01T02:13:18.022 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-01T02:13:18.024 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => { "msg": "Host mira063.front.sepia.ceph.com is running CentOS 7 (Core)" } 2019-10-01T02:13:18.076 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:18 +0000 (0:00:00.084) 0:02:19.411 ******* 2019-10-01T02:13:18.127 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-01T02:13:18.128 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-01T02:13:18.179 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:18 +0000 (0:00:00.105) 0:02:19.516 ******* 2019-10-01T02:13:18.216 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-01T02:13:18.218 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:18.277 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:18 +0000 (0:00:00.093) 0:02:19.610 ******* 2019-10-01T02:13:18.282 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-01T02:13:18.283 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-01T02:13:18.726 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-01T02:13:18.852 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-01T02:13:18.854 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.098320", "end": "2019-10-01 02:13:18.813430", "rc": 0, "start": "2019-10-01 02:13:18.715110", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-01T02:13:18.902 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:18 +0000 (0:00:00.629) 0:02:20.240 ******* 2019-10-01T02:13:18.950 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-01T02:13:18.953 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:18.988 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:18 +0000 (0:00:00.085) 0:02:20.325 ******* 2019-10-01T02:13:19.025 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-01T02:13:19.027 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.059 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.071) 0:02:20.397 ******* 2019-10-01T02:13:19.089 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.128 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.068) 0:02:20.465 ******* 2019-10-01T02:13:19.165 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-01T02:13:19.166 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-01T02:13:19.168 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.217 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.089) 0:02:20.555 ******* 2019-10-01T02:13:19.234 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-01T02:13:19.289 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.071) 0:02:20.626 ******* 2019-10-01T02:13:19.294 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-01T02:13:19.296 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.335 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.046) 0:02:20.672 ******* 2019-10-01T02:13:19.341 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-01T02:13:19.342 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => { "msg": "Host mira064.front.sepia.ceph.com is running CentOS 7 (Core)" } 2019-10-01T02:13:19.395 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.059) 0:02:20.732 ******* 2019-10-01T02:13:19.402 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-01T02:13:19.410 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.463 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.062) 0:02:20.794 ******* 2019-10-01T02:13:19.465 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-01T02:13:19.466 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-01T02:13:19.519 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.061) 0:02:20.856 ******* 2019-10-01T02:13:19.526 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-01T02:13:19.527 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:19.528 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:19.581 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.061) 0:02:20.918 ******* 2019-10-01T02:13:19.591 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-01T02:13:19.593 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.641 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.060) 0:02:20.978 ******* 2019-10-01T02:13:19.649 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-01T02:13:19.650 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.691 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.049) 0:02:21.028 ******* 2019-10-01T02:13:19.740 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-01T02:13:19.742 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.788 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.097) 0:02:21.125 ******* 2019-10-01T02:13:19.826 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-01T02:13:19.828 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:19.880 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.085) 0:02:21.211 ******* 2019-10-01T02:13:19.914 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-01T02:13:19.916 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:19.951 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:19 +0000 (0:00:00.077) 0:02:21.289 ******* 2019-10-01T02:13:19.980 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:20.027 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.075) 0:02:21.364 ******* 2019-10-01T02:13:20.064 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-01T02:13:20.065 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.102 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.075) 0:02:21.439 ******* 2019-10-01T02:13:20.134 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-10-01T02:13:20.135 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.205 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.103) 0:02:21.542 ******* 2019-10-01T02:13:20.213 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-01T02:13:20.215 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.088173", "end": "2019-10-01 02:13:20.118515", "rc": 0, "start": "2019-10-01 02:13:20.030342", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-01T02:13:20.252 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.045) 0:02:21.588 ******* 2019-10-01T02:13:20.255 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-01T02:13:20.257 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.295 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.045) 0:02:21.633 ******* 2019-10-01T02:13:20.303 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-01T02:13:20.305 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.340 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.043) 0:02:21.677 ******* 2019-10-01T02:13:20.346 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-01T02:13:20.348 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.384 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.044) 0:02:21.722 ******* 2019-10-01T02:13:20.390 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-01T02:13:20.392 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.426 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.041) 0:02:21.764 ******* 2019-10-01T02:13:20.434 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-01T02:13:20.434 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:20.472 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.045) 0:02:21.809 ******* 2019-10-01T02:13:20.478 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-01T02:13:20.479 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.516 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.042) 0:02:21.853 ******* 2019-10-01T02:13:20.523 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-01T02:13:20.527 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.572 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.053) 0:02:21.906 ******* 2019-10-01T02:13:20.576 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.611 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.042) 0:02:21.949 ******* 2019-10-01T02:13:20.616 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-01T02:13:20.618 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.656 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.044) 0:02:21.993 ******* 2019-10-01T02:13:20.663 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-01T02:13:20.665 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.712 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.055) 0:02:22.049 ******* 2019-10-01T02:13:20.718 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-01T02:13:20.719 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.774 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.061) 0:02:22.111 ******* 2019-10-01T02:13:20.783 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-01T02:13:20.785 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.826 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.052) 0:02:22.163 ******* 2019-10-01T02:13:20.827 INFO:teuthology.task.ansible.out: 2019-10-01T02:13:20.832 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-01T02:13:20.834 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.869 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.043) 0:02:22.207 ******* 2019-10-01T02:13:20.874 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-01T02:13:20.876 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:20.916 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.046) 0:02:22.253 ******* 2019-10-01T02:13:20.922 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-01T02:13:20.924 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:20.965 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:20 +0000 (0:00:00.048) 0:02:22.302 ******* 2019-10-01T02:13:20.973 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-01T02:13:20.974 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.017 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.051) 0:02:22.354 ******* 2019-10-01T02:13:21.022 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-01T02:13:21.029 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.065 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.048) 0:02:22.402 ******* 2019-10-01T02:13:21.071 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-01T02:13:21.072 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.107 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.042) 0:02:22.444 ******* 2019-10-01T02:13:21.111 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-01T02:13:21.112 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:21.151 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.043) 0:02:22.488 ******* 2019-10-01T02:13:21.168 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-01T02:13:21.170 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:21.203 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.052) 0:02:22.540 ******* 2019-10-01T02:13:21.211 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-01T02:13:21.253 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.049) 0:02:22.590 ******* 2019-10-01T02:13:21.263 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.296 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.043) 0:02:22.634 ******* 2019-10-01T02:13:21.302 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-01T02:13:21.304 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.356 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.058) 0:02:22.693 ******* 2019-10-01T02:13:21.363 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-10-01T02:13:21.365 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:21.415 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.060) 0:02:22.753 ******* 2019-10-01T02:13:21.456 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.506 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.090) 0:02:22.843 ******* 2019-10-01T02:13:21.545 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.585 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.079) 0:02:22.923 ******* 2019-10-01T02:13:21.625 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.662 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.076) 0:02:22.999 ******* 2019-10-01T02:13:21.701 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-01T02:13:21.702 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.767 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.104) 0:02:23.104 ******* 2019-10-01T02:13:21.808 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:21.852 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.085) 0:02:23.189 ******* 2019-10-01T02:13:21.859 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-01T02:13:21.860 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:13:21.918 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.063) 0:02:23.253 ******* 2019-10-01T02:13:21.924 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-01T02:13:21.928 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:21.962 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:21 +0000 (0:00:00.046) 0:02:23.299 ******* 2019-10-01T02:13:21.989 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-01T02:13:21.991 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.038 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.076) 0:02:23.376 ******* 2019-10-01T02:13:22.082 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.143 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.103) 0:02:23.479 ******* 2019-10-01T02:13:22.188 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.233 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.091) 0:02:23.570 ******* 2019-10-01T02:13:22.279 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.316 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.083) 0:02:23.653 ******* 2019-10-01T02:13:22.356 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.391 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.074) 0:02:23.728 ******* 2019-10-01T02:13:22.441 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-01T02:13:22.495 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.102) 0:02:23.831 ******* 2019-10-01T02:13:22.531 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-01T02:13:22.572 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.078) 0:02:23.909 ******* 2019-10-01T02:13:22.611 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-01T02:13:22.613 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:22.676 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:22 +0000 (0:00:00.104) 0:02:24.014 ******* 2019-10-01T02:13:23.140 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-01T02:13:23.142 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:13:23.175 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:23 +0000 (0:00:00.498) 0:02:24.513 ******* 2019-10-01T02:13:23.570 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-01T02:13:23.572 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-10-01T02:13:24.586 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-10-01T02:13:24.878 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-10-01T02:13:24.937 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:24 +0000 (0:00:01.760) 0:02:26.273 ******* 2019-10-01T02:13:25.942 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-10-01T02:13:26.015 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:26 +0000 (0:00:01.079) 0:02:27.352 ******* 2019-10-01T02:13:26.307 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-01T02:13:26.315 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4db37afba2545ac8871903964bcd8b701978565a", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 294, "state": "file", "uid": 0} 2019-10-01T02:13:27.345 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4db37afba2545ac8871903964bcd8b701978565a", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 294, "state": "file", "uid": 0} 2019-10-01T02:13:27.603 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "f1e1ffd80fcbd46e5168d3829bdcb34dffb3743b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 334, "state": "file", "uid": 0} 2019-10-01T02:13:27.647 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:27 +0000 (0:00:01.632) 0:02:28.984 ******* 2019-10-01T02:13:27.686 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:13:27.699 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-01T02:13:27.701 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:27.735 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:27 +0000 (0:00:00.087) 0:02:29.072 ******* 2019-10-01T02:13:27.782 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-01T02:13:27.784 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:27.825 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:27 +0000 (0:00:00.090) 0:02:29.162 ******* 2019-10-01T02:13:27.864 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-01T02:13:27.865 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:27.920 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:27 +0000 (0:00:00.095) 0:02:29.257 ******* 2019-10-01T02:13:27.960 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.007 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.086) 0:02:29.344 ******* 2019-10-01T02:13:28.045 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-01T02:13:28.046 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.101 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.093) 0:02:29.438 ******* 2019-10-01T02:13:28.140 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-01T02:13:28.142 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.184 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.084) 0:02:29.522 ******* 2019-10-01T02:13:28.211 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-01T02:13:28.213 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.250 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.065) 0:02:29.588 ******* 2019-10-01T02:13:28.279 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-01T02:13:28.281 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.321 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.070) 0:02:29.658 ******* 2019-10-01T02:13:28.360 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-01T02:13:28.362 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.398 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.076) 0:02:29.735 ******* 2019-10-01T02:13:28.684 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-01T02:13:28.685 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "f1e1ffd80fcbd46e5168d3829bdcb34dffb3743b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 334, "state": "file", "uid": 0} 2019-10-01T02:13:28.729 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.331) 0:02:30.066 ******* 2019-10-01T02:13:28.754 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:13:28.766 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-01T02:13:28.767 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.804 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.073) 0:02:30.140 ******* 2019-10-01T02:13:28.842 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-01T02:13:28.843 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.880 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.076) 0:02:30.217 ******* 2019-10-01T02:13:28.907 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-01T02:13:28.909 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:28.940 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:28 +0000 (0:00:00.060) 0:02:30.278 ******* 2019-10-01T02:13:28.977 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-01T02:13:28.979 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.014 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.073) 0:02:30.351 ******* 2019-10-01T02:13:29.041 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-01T02:13:29.042 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.088 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.073) 0:02:30.425 ******* 2019-10-01T02:13:29.116 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-01T02:13:29.117 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.162 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.073) 0:02:30.499 ******* 2019-10-01T02:13:29.189 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-01T02:13:29.190 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.225 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.062) 0:02:30.562 ******* 2019-10-01T02:13:29.252 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.297 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.072) 0:02:30.634 ******* 2019-10-01T02:13:29.326 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:29.359 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.062) 0:02:30.697 ******* 2019-10-01T02:13:29.535 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-10-01T02:13:29.536 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.696781", "end": "2019-10-01 02:13:29.500636", "rc": 0, "start": "2019-10-01 02:13:28.803855", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:13:29.582 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:29 +0000 (0:00:00.222) 0:02:30.919 ******* 2019-10-01T02:13:30.538 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.729303", "end": "2019-10-01 02:13:30.500576", "rc": 0, "start": "2019-10-01 02:13:29.771273", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:13:30.584 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:30 +0000 (0:00:01.002) 0:02:31.921 ******* 2019-10-01T02:13:30.721 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-01T02:13:30.722 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_6.x86_64 providing krb5-workstation is already installed"]} 2019-10-01T02:13:30.770 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:30 +0000 (0:00:00.186) 0:02:32.107 ******* 2019-10-01T02:13:30.815 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-01T02:13:30.816 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:30.856 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:30 +0000 (0:00:00.085) 0:02:32.193 ******* 2019-10-01T02:13:30.886 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-01T02:13:30.888 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:30.939 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:30 +0000 (0:00:00.083) 0:02:32.277 ******* 2019-10-01T02:13:30.979 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.032 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.092) 0:02:32.369 ******* 2019-10-01T02:13:31.061 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-01T02:13:31.062 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.107 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.074) 0:02:32.444 ******* 2019-10-01T02:13:31.355 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-01T02:13:31.356 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_6.x86_64 providing krb5-workstation is already installed"]} 2019-10-01T02:13:31.403 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.296) 0:02:32.740 ******* 2019-10-01T02:13:31.429 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-01T02:13:31.431 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.463 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.060) 0:02:32.801 ******* 2019-10-01T02:13:31.500 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-10-01T02:13:31.512 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.534 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.070) 0:02:32.871 ******* 2019-10-01T02:13:31.563 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-01T02:13:31.565 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.607 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.072) 0:02:32.944 ******* 2019-10-01T02:13:31.633 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-10-01T02:13:31.635 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:31.669 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:31 +0000 (0:00:00.062) 0:02:33.006 ******* 2019-10-01T02:13:32.364 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-10-01T02:13:32.424 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:32 +0000 (0:00:00.752) 0:02:33.758 ******* 2019-10-01T02:13:32.936 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-10-01T02:13:32.976 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:32 +0000 (0:00:00.554) 0:02:34.313 ******* 2019-10-01T02:13:46.795 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-10-01T02:13:46.795 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["smartmontools", "1:7.0-1.el7.x86_64 from base"]]}, "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.math.princeton.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package smartmontools.x86_64 1:6.5-1.el7 will be updated\n---> Package smartmontools.x86_64 1:7.0-1.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n smartmontools x86_64 1:7.0-1.el7 base 546 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package\n\nTotal download size: 546 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : 1:smartmontools-7.0-1.el7.x86_64 1/2 \n Cleanup : 1:smartmontools-6.5-1.el7.x86_64 2/2 \n Verifying : 1:smartmontools-7.0-1.el7.x86_64 1/2 \n Verifying : 1:smartmontools-6.5-1.el7.x86_64 2/2 \n\nUpdated:\n smartmontools.x86_64 1:7.0-1.el7 \n\nComplete!\n"]} 2019-10-01T02:13:46.829 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:46 +0000 (0:00:13.853) 0:02:48.167 ******* 2019-10-01T02:13:47.607 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["smartmontools", "1:7.0-1.el7.x86_64 from base"]]}, "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package smartmontools.x86_64 1:6.5-1.el7 will be updated\n---> Package smartmontools.x86_64 1:7.0-1.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n smartmontools x86_64 1:7.0-1.el7 base 546 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package\n\nTotal download size: 546 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : 1:smartmontools-7.0-1.el7.x86_64 1/2 \n Cleanup : 1:smartmontools-6.5-1.el7.x86_64 2/2 \n Verifying : 1:smartmontools-7.0-1.el7.x86_64 1/2 \n Verifying : 1:smartmontools-6.5-1.el7.x86_64 2/2 \n\nUpdated:\n smartmontools.x86_64 1:7.0-1.el7 \n\nComplete!\n"]} 2019-10-01T02:13:47.650 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:47 +0000 (0:00:00.821) 0:02:48.988 ******* 2019-10-01T02:13:48.150 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [mira063.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-10-01T02:13:48.948 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-10-01T02:13:49.404 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-10-01T02:13:50.177 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-10-01T02:13:50.606 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-10-01T02:13:50.652 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:50 +0000 (0:00:03.001) 0:02:51.990 ******* 2019-10-01T02:13:51.112 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-01T02:13:51.113 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-01T02:13:51.147 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:51 +0000 (0:00:00.494) 0:02:52.485 ******* 2019-10-01T02:13:51.417 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-01T02:13:51.418 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-10-01T02:13:51.473 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:51 +0000 (0:00:00.325) 0:02:52.810 ******* 2019-10-01T02:13:51.948 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-01T02:13:51.949 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-01T02:13:51.990 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:51 +0000 (0:00:00.517) 0:02:53.328 ******* 2019-10-01T02:13:52.430 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-01T02:13:52.431 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-10-01T02:13:53.221 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-10-01T02:13:53.707 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-10-01T02:13:54.409 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-10-01T02:13:54.995 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-10-01T02:13:55.042 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:03.051) 0:02:56.379 ******* 2019-10-01T02:13:55.072 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.125 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.082) 0:02:56.462 ******* 2019-10-01T02:13:55.164 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-01T02:13:55.165 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-10-01T02:13:55.210 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.085) 0:02:56.547 ******* 2019-10-01T02:13:55.622 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [mira064.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-10-01T02:13:55.657 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.446) 0:02:56.994 ******* 2019-10-01T02:13:55.664 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-01T02:13:55.665 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-01T02:13:55.703 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.043) 0:02:57.038 ******* 2019-10-01T02:13:55.707 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-01T02:13:55.709 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.747 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.045) 0:02:57.084 ******* 2019-10-01T02:13:55.752 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-01T02:13:55.754 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.765 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.796 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.798 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-01T02:13:55.799 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} TASK [common : Install nrpe without epel] ************************************** skipping: [mira063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:55.847 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.100) 0:02:57.184 ******* 2019-10-01T02:13:55.879 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:55 +0000 (0:00:00.031) 0:02:57.216 ******* 2019-10-01T02:13:56.353 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-01T02:13:56.355 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-10-01T02:13:56.387 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:56 +0000 (0:00:00.507) 0:02:57.724 ******* 2019-10-01T02:13:56.444 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-01T02:13:56.445 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:56.457 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:56.480 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:56.491 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-01T02:13:56.524 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:13:56 +0000 (0:00:00.137) 0:02:57.861 ******* 2019-10-01T02:14:02.262 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-01T02:14:02.263 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-common", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nagios-common are up to date", ""]} 2019-10-01T02:14:03.021 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-common", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nagios-common are up to date", ""]} 2019-10-01T02:14:08.597 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nrpe", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nrpe are up to date", ""]} 2019-10-01T02:14:09.456 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nrpe", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nrpe are up to date", ""]} 2019-10-01T02:14:19.839 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": [], "updated": [["nagios-plugins", "2.2.1-17.20190829gitfb792ff.el7.x86_64 from epel"]]}, "item": "nagios-plugins", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.math.princeton.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package nagios-plugins.x86_64 0:2.2.1-16.20180725git3429dad.el7 will be updated\n--> Processing Dependency: nagios-plugins = 2.2.1-16.20180725git3429dad.el7 for package: nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64\n---> Package nagios-plugins.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 will be an update\n--> Running transaction check\n---> Package nagios-plugins-load.x86_64 0:2.2.1-16.20180725git3429dad.el7 will be updated\n---> Package nagios-plugins-load.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository\n Size\n================================================================================\nUpdating:\n nagios-plugins x86_64 2.2.1-17.20190829gitfb792ff.el7 epel 114 k\nUpdating for dependencies:\n nagios-plugins-load x86_64 2.2.1-17.20190829gitfb792ff.el7 epel 40 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTot 2019-10-01T02:14:19.841 INFO:teuthology.task.ansible.out:al download size: 155 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 275 kB/s | 155 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : nagios-plugins-2.2.1-17.20190829gitfb792ff.el7.x86_64 1/4 \n Updating : nagios-plugins-load-2.2.1-17.20190829gitfb792ff.el7.x86_64 2/4 \n Cleanup : nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64 3/4 \n Cleanup : nagios-plugins-2.2.1-16.20180725git3429dad.el7.x86_64 4/4 \n Verifying : nagios-plugins-2.2.1-17.20190829gitfb792ff.el7.x86_64 1/4 \n Verifying : nagios-plugins-load-2.2.1-17.20190829gitfb792ff.el7.x86_64 2/4 \n Verifying : nagios-plugins-2.2.1-16.20180725git3429dad.el7.x86_64 3/4 \n Verifying : nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64 4/4 \n\nUpdated:\n nagios-plugins.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 \n\nDependency Updated:\n nagios-plugins-load.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 \n\nComplete!\n"]} 2019-10-01T02:14:20.406 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": [], "updated": [["nagios-plugins", "2.2.1-17.20190829gitfb792ff.el7.x86_64 from epel"]]}, "item": "nagios-plugins", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package nagios-plugins.x86_64 0:2.2.1-16.20180725git3429dad.el7 will be updated\n--> Processing Dependency: nagios-plugins = 2.2.1-16.20180725git3429dad.el7 for package: nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64\n---> Package nagios-plugins.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 will be an update\n--> Running transaction check\n---> Package nagios-plugins-load.x86_64 0:2.2.1-16.20180725git3429dad.el7 will be updated\n---> Package nagios-plugins-load.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository\n Size\n================================================================================\nUpdating:\n nagios-plugins x86_64 2.2.1-17.20190829gitfb792ff.el7 epel 114 k\nUpdating for dependencies:\n nagios-plugins-load x86_64 2.2.1-17.20190829gitfb792ff.el7 epel 40 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal dow 2019-10-01T02:14:20.408 INFO:teuthology.task.ansible.out:nload size: 155 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 348 kB/s | 155 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : nagios-plugins-2.2.1-17.20190829gitfb792ff.el7.x86_64 1/4 \n Updating : nagios-plugins-load-2.2.1-17.20190829gitfb792ff.el7.x86_64 2/4 \n Cleanup : nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64 3/4 \n Cleanup : nagios-plugins-2.2.1-16.20180725git3429dad.el7.x86_64 4/4 \n Verifying : nagios-plugins-2.2.1-17.20190829gitfb792ff.el7.x86_64 1/4 \n Verifying : nagios-plugins-load-2.2.1-17.20190829gitfb792ff.el7.x86_64 2/4 \n Verifying : nagios-plugins-2.2.1-16.20180725git3429dad.el7.x86_64 3/4 \n Verifying : nagios-plugins-load-2.2.1-16.20180725git3429dad.el7.x86_64 4/4 \n\nUpdated:\n nagios-plugins.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 \n\nDependency Updated:\n nagios-plugins-load.x86_64 0:2.2.1-17.20190829gitfb792ff.el7 \n\nComplete!\n"]} 2019-10-01T02:14:26.200 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins-load", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nagios-plugins-load are up to date", ""]} 2019-10-01T02:14:26.237 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:26 +0000 (0:00:29.713) 0:03:27.575 ******* 2019-10-01T02:14:26.281 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-01T02:14:26.283 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-common", "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "item": "nrpe", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.293 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.308 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-load", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.345 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:26 +0000 (0:00:00.107) 0:03:27.682 ******* 2019-10-01T02:14:26.766 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-01T02:14:26.766 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": [], "updated": []}, "item": "nagios-plugins-load", "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["All packages providing nagios-plugins-load are up to date", ""]} 2019-10-01T02:14:26.808 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:26 +0000 (0:00:00.462) 0:03:28.145 ******* 2019-10-01T02:14:26.851 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-01T02:14:26.851 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=nagios-common) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-common", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.863 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=nrpe) => {"ansible_loop_var": "item", "changed": false, "item": "nrpe", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.875 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=nagios-plugins) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.876 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=nagios-plugins-load) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-load", "skip_reason": "Conditional result was False"} 2019-10-01T02:14:26.917 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:26 +0000 (0:00:00.109) 0:03:28.255 ******* 2019-10-01T02:14:27.598 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-01T02:14:27.599 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-10-01T02:14:27.637 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:27 +0000 (0:00:00.719) 0:03:28.975 ******* 2019-10-01T02:14:28.217 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-10-01T02:14:28.263 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:28 +0000 (0:00:00.624) 0:03:29.600 ******* 2019-10-01T02:14:28.877 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-01T02:14:28.877 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-10-01T02:14:28.912 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:28 +0000 (0:00:00.648) 0:03:30.249 ******* 2019-10-01T02:14:28.940 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:14:28.976 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:28 +0000 (0:00:00.065) 0:03:30.314 ******* 2019-10-01T02:14:29.422 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:14:29.459 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:29 +0000 (0:00:00.482) 0:03:30.796 ******* 2019-10-01T02:14:29.531 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-01T02:14:29.533 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-10-01T02:14:29.577 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:29 +0000 (0:00:00.117) 0:03:30.914 ******* 2019-10-01T02:14:29.603 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-01T02:14:29.605 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:14:29.642 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:29 +0000 (0:00:00.065) 0:03:30.980 ******* 2019-10-01T02:14:29.964 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-01T02:14:29.965 INFO:teuthology.task.ansible.out:fatal: [mira063.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-10-01T02:14:29.998 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:29 +0000 (0:00:00.355) 0:03:31.335 ******* 2019-10-01T02:14:30.024 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-01T02:14:30.026 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:14:30.059 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:30 +0000 (0:00:00.060) 0:03:31.396 ******* 2019-10-01T02:14:30.117 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:14:30.151 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:30 +0000 (0:00:00.092) 0:03:31.489 ******* 2019-10-01T02:14:30.670 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-01T02:14:30.672 INFO:teuthology.task.ansible.out:fatal: [mira064.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-10-01T02:14:30.704 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:30 +0000 (0:00:00.553) 0:03:32.042 ******* 2019-10-01T02:14:30.740 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-01T02:14:30.741 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:14:30.774 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:30 +0000 (0:00:00.069) 0:03:32.111 ******* 2019-10-01T02:14:31.321 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-10-01T02:14:31.322 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-10-01T02:14:31.357 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:31 +0000 (0:00:00.582) 0:03:32.694 ******* 2019-10-01T02:14:32.012 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-10-01T02:14:32.049 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:32 +0000 (0:00:00.692) 0:03:33.387 ******* 2019-10-01T02:14:32.440 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2019-10-01 02:10:17 UTC", "ActiveEnterTimestampMonotonic": "24096809", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target var-run.mount systemd-journald.socket time-sync.target local-fs.target network.target nss-lookup.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Tue 2019-10-01 02:10:17 UTC", "AssertTimestampMonotonic": "24095850", "Before": "multi-user.target xdm.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2019-10-01 02:10:17 UTC", "ConditionTimestampMonotonic": "24095850", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "4484", "ExecMainStartTimestamp": "Tue 2019-10-01 02:10:17 UTC", "ExecMainStartTimestampMonotonic": "24096724", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/ 2019-10-01T02:14:32.442 INFO:teuthology.task.ansible.out:nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2019-10-01 02:10:17 UTC", "InactiveExitTimestampMonotonic": "24096809", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "4484", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target network-online.target", "Restart": "no", "RestartUSec" 2019-10-01T02:14:32.444 INFO:teuthology.task.ansible.out:: "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Tue 2019-10-01 02:10:17 UTC", "WatchdogTimestampMonotonic": "24096776", "WatchdogUSec": "0"}} 2019-10-01T02:14:32.480 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:32 +0000 (0:00:00.430) 0:03:33.818 ******* 2019-10-01T02:14:32.691 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2019-10-01 02:10:23 UTC", "ActiveEnterTimestampMonotonic": "22025630", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount network.target local-fs.target time-sync.target nss-lookup.target systemd-journald.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Tue 2019-10-01 02:10:23 UTC", "AssertTimestampMonotonic": "22024403", "Before": "getty@tty1.service shutdown.target xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2019-10-01 02:10:23 UTC", "ConditionTimestampMonotonic": "22024403", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "4435", "ExecMainStartTimestamp": "Tue 2019-10-01 02:10:23 UTC", "ExecMainStartTimestampMonotonic": "22025524", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_error 2019-10-01T02:14:32.693 INFO:teuthology.task.ansible.out:s=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2019-10-01 02:10:23 UTC", "InactiveExitTimestampMonotonic": "22025630", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "4435", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMod 2019-10-01T02:14:32.694 INFO:teuthology.task.ansible.out:e": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Tue 2019-10-01 02:10:23 UTC", "WatchdogTimestampMonotonic": "22025586", "WatchdogUSec": "0"}} 2019-10-01T02:14:32.727 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:32 +0000 (0:00:00.246) 0:03:34.064 ******* 2019-10-01T02:14:32.932 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.005846", "end": "2019-10-01 02:14:32.894741", "rc": 0, "start": "2019-10-01 02:14:32.888895", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-10-01T02:14:32.967 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:32 +0000 (0:00:00.240) 0:03:34.304 ******* 2019-10-01T02:14:33.189 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.005539", "end": "2019-10-01 02:14:33.153371", "rc": 0, "start": "2019-10-01 02:14:33.147832", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-10-01T02:14:33.231 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:33 +0000 (0:00:00.264) 0:03:34.569 ******* 2019-10-01T02:14:33.698 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-01T02:14:33.698 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed"]} 2019-10-01T02:14:33.944 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed"]} 2019-10-01T02:14:34.399 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "msg": "", "rc": 0, "results": ["policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-10-01T02:14:34.438 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:34 +0000 (0:00:01.206) 0:03:35.775 ******* 2019-10-01T02:14:34.685 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "msg": "", "rc": 0, "results": ["policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-10-01T02:14:34.721 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:14:34 +0000 (0:00:00.283) 0:03:36.059 ******* 2019-10-01T02:15:13.161 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-01T02:15:13.162 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7.1.noarch from updates"]]}, "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.math.princeton.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-229.el7_6.12 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7.1 for package: selinux-policy-targeted-3.13.1-252.el7.1.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7.1 for package: selinux-policy-targeted-3.13.1-252.el7.1.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-229.el7_6.12 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7.1 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7.1 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not install 2019-10-01T02:15:13.163 INFO:teuthology.task.ansible.out:ed.\n--------------------------------------------------------------------------------\nTotal 23 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7.1.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7.1.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-229.el7_6.12.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-229.el7_6.12.noarch 4/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 1/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-229.el7_6.12.noarch 3/4 \n Verifying : selinux-policy-3.13.1-229.el7_6.12.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7.1 \n\nComplete!\n"]} 2019-10-01T02:15:13.205 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:13 +0000 (0:00:38.483) 0:04:14.543 ******* 2019-10-01T02:15:13.631 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7.1.noarch from updates"]]}, "msg": "", "obsoletes": {"urw-fonts": {"dist": "noarch", "repo": "@base", "version": "2.4-16.el7"}}, "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.facebook.net\n * epel: ftp.linux.ncsu.edu\n * extras: mirrors.mit.edu\n * updates: mirror.ash.fastserv.com\n1 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-229.el7_6.12 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7.1 for package: selinux-policy-targeted-3.13.1-252.el7.1.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7.1 for package: selinux-policy-targeted-3.13.1-252.el7.1.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-229.el7_6.12 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7.1 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7.1 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n- 2019-10-01T02:15:13.633 INFO:teuthology.task.ansible.out:-------------------------------------------------------------------------------\nTotal 24 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7.1.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7.1.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-229.el7_6.12.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-229.el7_6.12.noarch 4/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 1/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-229.el7_6.12.noarch 3/4 \n Verifying : selinux-policy-3.13.1-229.el7_6.12.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7.1 \n\nComplete!\n"]} 2019-10-01T02:15:13.664 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:13 +0000 (0:00:00.458) 0:04:15.001 ******* 2019-10-01T02:15:14.263 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-01T02:15:14.265 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-10-01T02:15:14.300 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:14 +0000 (0:00:00.635) 0:04:15.637 ******* 2019-10-01T02:15:14.945 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-10-01T02:15:14.981 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:14 +0000 (0:00:00.681) 0:04:16.318 ******* 2019-10-01T02:15:28.801 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-01T02:15:28.803 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.047245", "end": "2019-10-01 02:15:28.765780", "failed_when_result": false, "rc": 0, "start": "2019-10-01 02:15:14.718535", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-10-01T02:15:28.851 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:28 +0000 (0:00:13.869) 0:04:30.188 ******* 2019-10-01T02:15:29.670 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.240350", "end": "2019-10-01 02:15:29.623544", "failed_when_result": false, "rc": 0, "start": "2019-10-01 02:15:15.383194", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-10-01T02:15:29.710 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:29 +0000 (0:00:00.859) 0:04:31.048 ******* 2019-10-01T02:15:30.012 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-10-01T02:15:30.054 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:30 +0000 (0:00:00.344) 0:04:31.392 ******* 2019-10-01T02:15:30.539 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014196", "end": "2019-10-01 02:15:30.498627", "rc": 0, "start": "2019-10-01 02:15:30.484431", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-10-01T02:15:30.586 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:30 +0000 (0:00:00.531) 0:04:31.923 ******* 2019-10-01T02:15:31.012 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-01T02:15:31.013 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-10-01T02:15:31.067 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:31 +0000 (0:00:00.480) 0:04:32.404 ******* 2019-10-01T02:15:31.072 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-01T02:15:31.073 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012285", "end": "2019-10-01 02:15:31.012199", "rc": 0, "start": "2019-10-01 02:15:30.999914", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:15:31.112 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:31 +0000 (0:00:00.040) 0:04:32.445 ******* 2019-10-01T02:15:31.480 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-01T02:15:31.481 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.021957", "end": "2019-10-01 02:15:31.449164", "rc": 0, "start": "2019-10-01 02:15:31.427207", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-10-01T02:15:31.530 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:31 +0000 (0:00:00.422) 0:04:32.868 ******* 2019-10-01T02:15:31.922 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-01T02:15:31.923 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012885", "end": "2019-10-01 02:15:31.885079", "rc": 0, "start": "2019-10-01 02:15:31.872194", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:15:31.960 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:31 +0000 (0:00:00.429) 0:04:33.297 ******* 2019-10-01T02:15:31.961 INFO:teuthology.task.ansible.out: 2019-10-01T02:15:45.723 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-01T02:15:45.724 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.153571", "end": "2019-10-01 02:15:45.680631", "rc": 0, "start": "2019-10-01 02:15:31.527060", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:15:45.758 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:45 +0000 (0:00:13.798) 0:04:47.096 ******* 2019-10-01T02:15:46.240 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-10-01T02:15:46.273 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.514) 0:04:47.610 ******* 2019-10-01T02:15:46.304 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-01T02:15:46.305 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-10-01T02:15:46.341 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.067) 0:04:47.678 ******* 2019-10-01T02:15:46.389 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-01T02:15:46.391 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-01T02:15:46.431 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.089) 0:04:47.768 ******* 2019-10-01T02:15:46.479 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-01T02:15:46.481 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-10-01T02:15:46.521 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.088) 0:04:47.857 ******* 2019-10-01T02:15:46.570 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-01T02:15:46.571 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-10-01T02:15:46.608 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.088) 0:04:47.945 ******* 2019-10-01T02:15:46.868 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-01T02:15:46.869 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.462615", "end": "2019-10-01 02:15:46.824253", "rc": 0, "start": "2019-10-01 02:15:32.361638", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:15:46.908 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:46 +0000 (0:00:00.299) 0:04:48.245 ******* 2019-10-01T02:15:47.068 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-01T02:15:47.123 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.214) 0:04:48.460 ******* 2019-10-01T02:15:47.124 INFO:teuthology.task.ansible.out: 2019-10-01T02:15:47.389 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-01T02:15:47.389 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-10-01T02:15:47.434 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.310) 0:04:48.771 ******* 2019-10-01T02:15:47.471 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-01T02:15:47.472 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-10-01T02:15:47.507 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.073) 0:04:48.844 ******* 2019-10-01T02:15:47.557 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-01T02:15:47.557 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-01T02:15:47.617 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.102) 0:04:48.947 ******* 2019-10-01T02:15:47.619 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-01T02:15:47.620 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 1206, "name": "ubuntu", "state": "present", "system": false} 2019-10-01T02:15:47.668 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.058) 0:04:49.005 ******* 2019-10-01T02:15:47.676 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-01T02:15:47.677 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-10-01T02:15:47.718 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.050) 0:04:49.056 ******* 2019-10-01T02:15:47.724 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-01T02:15:47.725 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:15:47.759 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.040) 0:04:49.096 ******* 2019-10-01T02:15:47.780 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [mira064.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-10-01T02:15:47.792 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. TASK [testnode : Delete the teuthology users password.] ************************ skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:15:47.832 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.068) 0:04:49.165 ******* 2019-10-01T02:15:47.858 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:47 +0000 (0:00:00.030) 0:04:49.195 ******* 2019-10-01T02:15:48.314 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-01T02:15:48.315 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-10-01T02:15:48.354 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:48 +0000 (0:00:00.496) 0:04:49.691 ******* 2019-10-01T02:15:48.360 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-01T02:15:48.364 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-10-01T02:15:48.407 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:48 +0000 (0:00:00.053) 0:04:49.745 ******* 2019-10-01T02:15:48.819 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-01T02:15:48.820 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 1206, "name": "ubuntu", "state": "present", "system": false} 2019-10-01T02:15:48.854 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:48 +0000 (0:00:00.446) 0:04:50.191 ******* 2019-10-01T02:15:48.881 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-01T02:15:48.883 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:15:48.919 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:48 +0000 (0:00:00.065) 0:04:50.256 ******* 2019-10-01T02:15:48.938 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:15:48.951 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:15:48.985 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:48 +0000 (0:00:00.065) 0:04:50.322 ******* 2019-10-01T02:15:49.494 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-01T02:15:49.496 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-10-01T02:15:49.548 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:49 +0000 (0:00:00.562) 0:04:50.884 ******* 2019-10-01T02:15:49.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-10-01T02:15:49.842 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:49 +0000 (0:00:00.294) 0:04:51.179 ******* 2019-10-01T02:15:49.882 INFO:teuthology.task.ansible.out: 2019-10-01T02:15:49.884 INFO:teuthology.task.ansible.out:TASK [testnode : Set hostname var] ********************************************* ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "mira063"}, "changed": false} 2019-10-01T02:15:49.922 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:49 +0000 (0:00:00.079) 0:04:51.259 ******* 2019-10-01T02:15:50.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-10-01T02:15:50.994 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:50 +0000 (0:00:01.072) 0:04:52.331 ******* 2019-10-01T02:15:51.032 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-01T02:15:51.034 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "mira064"}, "changed": false} 2019-10-01T02:15:51.072 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:51 +0000 (0:00:00.078) 0:04:52.410 ******* 2019-10-01T02:15:51.131 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-01T02:15:51.132 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "mira063.front.sepia.ceph.com", "ansible_hostname": "mira063", "ansible_nodename": "mira063"}, "changed": true, "name": "mira063"} 2019-10-01T02:15:51.172 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:51 +0000 (0:00:00.099) 0:04:52.510 ******* 2019-10-01T02:15:51.802 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "mira064.front.sepia.ceph.com", "ansible_hostname": "mira064", "ansible_nodename": "mira064"}, "changed": true, "name": "mira064"} 2019-10-01T02:15:51.837 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:51 +0000 (0:00:00.664) 0:04:53.174 ******* 2019-10-01T02:15:51.838 INFO:teuthology.task.ansible.out: 2019-10-01T02:15:52.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-10-01T02:15:52.417 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-10-01T02:15:52.460 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:52 +0000 (0:00:00.621) 0:04:53.796 ******* 2019-10-01T02:15:52.946 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-10-01T02:15:52.986 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:52 +0000 (0:00:00.527) 0:04:54.324 ******* 2019-10-01T02:15:53.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-10-01T02:15:53.763 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:53 +0000 (0:00:00.775) 0:04:55.099 ******* 2019-10-01T02:15:54.195 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-10-01T02:15:54.229 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:54 +0000 (0:00:00.467) 0:04:55.566 ******* 2019-10-01T02:15:55.103 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:15:55.116 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-01T02:15:55.118 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [mira063.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-01T02:15:55.151 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:55 +0000 (0:00:00.922) 0:04:56.489 ******* 2019-10-01T02:15:55.547 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:15:55.568 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-01T02:15:55.610 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:55 +0000 (0:00:00.458) 0:04:56.947 ******* 2019-10-01T02:15:56.425 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-10-01T02:15:56.458 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:56 +0000 (0:00:00.846) 0:04:57.794 ******* 2019-10-01T02:15:56.850 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-10-01T02:15:56.900 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:56 +0000 (0:00:00.442) 0:04:58.237 ******* 2019-10-01T02:15:56.928 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-01T02:15:56.969 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:56 +0000 (0:00:00.069) 0:04:58.306 ******* 2019-10-01T02:15:57.378 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-01T02:15:57.427 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:57 +0000 (0:00:00.458) 0:04:58.764 ******* 2019-10-01T02:15:57.436 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-01T02:15:57.437 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2019-10-01T02:15:57.474 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:57 +0000 (0:00:00.047) 0:04:58.812 ******* 2019-10-01T02:15:57.902 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2019-10-01T02:15:57.939 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:57 +0000 (0:00:00.464) 0:04:59.276 ******* 2019-10-01T02:15:57.998 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-01T02:15:58.000 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1206, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-01T02:15:58.038 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:58 +0000 (0:00:00.098) 0:04:59.375 ******* 2019-10-01T02:15:58.459 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1206, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-01T02:15:58.495 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:58 +0000 (0:00:00.457) 0:04:59.832 ******* 2019-10-01T02:15:59.398 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-10-01T02:15:59.401 INFO:teuthology.task.ansible.out: 2019-10-01T02:15:59.440 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:59 +0000 (0:00:00.945) 0:05:00.777 ******* 2019-10-01T02:15:59.857 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-10-01T02:15:59.896 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:15:59 +0000 (0:00:00.456) 0:05:01.234 ******* 2019-10-01T02:16:00.763 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-01T02:16:00.764 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-10-01T02:16:00.828 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:00 +0000 (0:00:00.931) 0:05:02.165 ******* 2019-10-01T02:16:01.322 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-10-01T02:16:01.364 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:01 +0000 (0:00:00.536) 0:05:02.701 ******* 2019-10-01T02:16:03.733 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-01T02:16:03.734 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-10-01T02:16:03.735 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.433926", "end": "2019-10-01 02:16:03.693943", "rc": 0, "start": "2019-10-01 02:16:01.260017", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:16:03.778 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:03 +0000 (0:00:02.413) 0:05:05.115 ******* 2019-10-01T02:16:03.933 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-01T02:16:04.089 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:04 +0000 (0:00:00.307) 0:05:05.422 ******* 2019-10-01T02:16:04.562 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-01T02:16:04.563 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.721027", "end": "2019-10-01 02:16:04.515557", "rc": 0, "start": "2019-10-01 02:16:01.794530", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:16:04.600 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:04 +0000 (0:00:00.514) 0:05:05.937 ******* 2019-10-01T02:16:04.637 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-01T02:16:04.672 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:04 +0000 (0:00:00.072) 0:05:06.010 ******* 2019-10-01T02:16:05.689 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-01T02:16:05.690 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "md5sum": "1ab983587ffdee6ea400c98696efa07e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569896164.13-196678415989675/source", "state": "file", "uid": 0} 2019-10-01T02:16:06.220 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "md5sum": "1ab983587ffdee6ea400c98696efa07e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569896164.72-162733270545720/source", "state": "file", "uid": 0} 2019-10-01T02:16:07.209 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "md5sum": "20e0446046d8fc74984ed9140203c656", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569896165.71-73280833053821/source", "state": "file", "uid": 0} 2019-10-01T02:16:07.242 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:02.569) 0:05:08.580 ******* 2019-10-01T02:16:07.270 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-01T02:16:07.271 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.308 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:00.065) 0:05:08.645 ******* 2019-10-01T02:16:07.346 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-01T02:16:07.348 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.359 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.394 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:00.086) 0:05:08.731 ******* 2019-10-01T02:16:07.423 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:16:07.713 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "md5sum": "20e0446046d8fc74984ed9140203c656", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1569896166.23-241011812417038/source", "state": "file", "uid": 0} 2019-10-01T02:16:07.753 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:00.359) 0:05:09.091 ******* 2019-10-01T02:16:07.781 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-01T02:16:07.783 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.821 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:00.066) 0:05:09.157 ******* 2019-10-01T02:16:07.859 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-01T02:16:07.861 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.870 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:16:07.908 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:07 +0000 (0:00:00.087) 0:05:09.245 ******* 2019-10-01T02:16:07.937 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:16:08.273 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.404692", "end": "2019-10-01 02:16:08.228575", "rc": 0, "start": "2019-10-01 02:16:07.823883", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nCleaning up list of fastest mirrors", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Cleaning up list of fastest mirrors"]} 2019-10-01T02:16:08.314 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:08 +0000 (0:00:00.406) 0:05:09.651 ******* 2019-10-01T02:16:08.371 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:16:08.373 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-10-01T02:16:08.411 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:08 +0000 (0:00:00.096) 0:05:09.748 ******* 2019-10-01T02:16:08.797 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.410520", "end": "2019-10-01 02:16:08.759130", "rc": 0, "start": "2019-10-01 02:16:08.348610", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nCleaning up list of fastest mirrors", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Cleaning up list of fastest mirrors"]} 2019-10-01T02:16:08.849 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:08 +0000 (0:00:00.437) 0:05:10.186 ******* 2019-10-01T02:16:08.892 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-10-01T02:16:08.923 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:08 +0000 (0:00:00.075) 0:05:10.261 ******* 2019-10-01T02:16:09.084 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* ok: [mira063.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-10-01T02:16:09.131 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:09 +0000 (0:00:00.204) 0:05:10.465 ******* 2019-10-01T02:16:09.649 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-10-01T02:16:09.705 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:09 +0000 (0:00:00.576) 0:05:11.042 ******* 2019-10-01T02:16:14.697 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:05.165522", "end": "2019-10-01 02:16:14.647974", "rc": 0, "start": "2019-10-01 02:16:09.482452", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.vtti.vt.edu", " * epel: ftp.linux.ncsu.edu", " * extras: mirror.fileplanet.com", " * updates: mirror.jaleco.com", "No unfinished transactions left."]} 2019-10-01T02:16:14.752 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:14 +0000 (0:00:05.046) 0:05:16.088 ******* 2019-10-01T02:16:15.266 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-01T02:16:15.267 INFO:teuthology.task.ansible.out:fatal: [mira063.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.089005", "end": "2019-10-01 02:16:15.231497", "msg": "non-zero return code", "rc": 1, "start": "2019-10-01 02:16:15.142492", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-10-01T02:16:15.300 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:15 +0000 (0:00:00.549) 0:05:16.638 ******* 2019-10-01T02:16:15.338 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-01T02:16:15.340 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:16:15.384 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:15 +0000 (0:00:00.083) 0:05:16.721 ******* 2019-10-01T02:16:15.545 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-01T02:16:15.547 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:05.351958", "end": "2019-10-01 02:16:15.494302", "rc": 0, "start": "2019-10-01 02:16:10.142344", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.vtti.vt.edu", " * epel: ftp.linux.ncsu.edu", " * extras: repo1.ash.innoscale.net", " * updates: mirror.jaleco.com", "No unfinished transactions left."]} 2019-10-01T02:16:15.603 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:15 +0000 (0:00:00.219) 0:05:16.941 ******* 2019-10-01T02:16:16.066 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ ok: [mira063.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "msg": "", "rc": 0, "results": ["ceph is not installed"]} 2019-10-01T02:16:16.098 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-01T02:16:16.099 INFO:teuthology.task.ansible.out:fatal: [mira064.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.056894", "end": "2019-10-01 02:16:16.063761", "msg": "non-zero return code", "rc": 1, "start": "2019-10-01 02:16:16.006867", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-10-01T02:16:16.145 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:16 +0000 (0:00:00.541) 0:05:17.482 ******* 2019-10-01T02:16:16.195 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-01T02:16:16.197 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:16:16.253 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:16 +0000 (0:00:00.107) 0:05:17.590 ******* 2019-10-01T02:16:16.792 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ ok: [mira063.front.sepia.ceph.com] => (item=ceph-base) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-base", "msg": "", "rc": 0, "results": ["ceph-base is not installed"]} 2019-10-01T02:16:16.884 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "msg": "", "rc": 0, "results": ["ceph is not installed"]} 2019-10-01T02:16:17.506 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ceph-selinux) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-selinux", "msg": "", "rc": 0, "results": ["ceph-selinux is not installed"]} 2019-10-01T02:16:17.657 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-base) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-base", "msg": "", "rc": 0, "results": ["ceph-base is not installed"]} 2019-10-01T02:16:18.308 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "msg": "", "rc": 0, "results": ["ceph-common is not installed"]} 2019-10-01T02:16:18.450 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-selinux) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-selinux", "msg": "", "rc": 0, "results": ["ceph-selinux is not installed"]} 2019-10-01T02:16:19.114 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ceph-debuginfo) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-debuginfo", "msg": "", "rc": 0, "results": ["ceph-debuginfo is not installed"]} 2019-10-01T02:16:19.240 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "msg": "", "rc": 0, "results": ["ceph-common is not installed"]} 2019-10-01T02:16:19.850 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ceph-release) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-release", "msg": "", "rc": 0, "results": ["ceph-release is not installed"]} 2019-10-01T02:16:20.017 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-debuginfo) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-debuginfo", "msg": "", "rc": 0, "results": ["ceph-debuginfo is not installed"]} 2019-10-01T02:16:20.443 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "msg": "", "rc": 0, "results": ["libcephfs1 is not installed"]} 2019-10-01T02:16:20.816 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-release) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-release", "msg": "", "rc": 0, "results": ["ceph-release is not installed"]} 2019-10-01T02:16:21.250 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ceph-radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-radosgw", "msg": "", "rc": 0, "results": ["ceph-radosgw is not installed"]} 2019-10-01T02:16:21.445 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "msg": "", "rc": 0, "results": ["libcephfs1 is not installed"]} 2019-10-01T02:16:22.073 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-01T02:16:22.284 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ceph-radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-radosgw", "msg": "", "rc": 0, "results": ["ceph-radosgw is not installed"]} 2019-10-01T02:16:22.874 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "msg": "", "rc": 0, "results": ["python-rados is not installed"]} 2019-10-01T02:16:23.080 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-01T02:16:23.587 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "msg": "", "rc": 0, "results": ["python-rbd is not installed"]} 2019-10-01T02:16:23.858 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "msg": "", "rc": 0, "results": ["python-rados is not installed"]} 2019-10-01T02:16:24.362 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "msg": "", "rc": 0, "results": ["python-cephfs is not installed"]} 2019-10-01T02:16:24.664 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "msg": "", "rc": 0, "results": ["python-rbd is not installed"]} 2019-10-01T02:16:25.428 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "msg": "", "rc": 0, "results": ["python-cephfs is not installed"]} 2019-10-01T02:16:31.927 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librbd1"]}, "item": "librbd1", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-img-1.5.3-160.el7_6.2.x86_64\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-160.el7_6.2 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-160.el7_6.2 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-160.el7_6.2 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librbd1 x86_64 1:10.2.5-4.el7 @base 7.4 M\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-160.el7_6.2 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-160.el7_6.2 @updates 5.8 M\n qemu-kvm-common x86_64 10:1.5.3-160.el7_6.2 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+3 Dependent packages)\n\nInstalled size: 16 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64 1/4 \n Erasing : 10:qemu-img-1.5.3-160.el7_6.2.x 2019-10-01T02:16:31.929 INFO:teuthology.task.ansible.out:86_64 2/4 \n Erasing : 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64 3/4 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/4 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/4 \n Verifying : 10:qemu-img-1.5.3-160.el7_6.2.x86_64 2/4 \n Verifying : 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64 3/4 \n Verifying : 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64 4/4 \n\nRemoved:\n librbd1.x86_64 1:10.2.5-4.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-160.el7_6.2 \n qemu-kvm.x86_64 10:1.5.3-160.el7_6.2 \n qemu-kvm-common.x86_64 10:1.5.3-160.el7_6.2 \n\nComplete!\n"]} 2019-10-01T02:16:34.409 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librados2"]}, "item": "librados2", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 5.5 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 \n\nComplete!\n"]} 2019-10-01T02:16:35.905 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librbd1"]}, "item": "librbd1", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-img-1.5.3-160.el7_6.2.x86_64\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-160.el7_6.2 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-160.el7_6.2 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-160.el7_6.2 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librbd1 x86_64 1:10.2.5-4.el7 @base 7.4 M\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-160.el7_6.2 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-160.el7_6.2 @updates 5.8 M\n qemu-kvm-common x86_64 10:1.5.3-160.el7_6.2 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+3 Dependent packages)\n\nInstalled size: 16 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64 1/4 \n Erasing : 10:qemu-img-1.5.3-160.el7_6.2.x 2019-10-01T02:16:35.906 INFO:teuthology.task.ansible.out:86_64 2/4 \n Erasing : 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64 3/4 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/4 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/4 \n Verifying : 10:qemu-img-1.5.3-160.el7_6.2.x86_64 2/4 \n Verifying : 10:qemu-kvm-1.5.3-160.el7_6.2.x86_64 3/4 \n Verifying : 10:qemu-kvm-common-1.5.3-160.el7_6.2.x86_64 4/4 \n\nRemoved:\n librbd1.x86_64 1:10.2.5-4.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-160.el7_6.2 \n qemu-kvm.x86_64 10:1.5.3-160.el7_6.2 \n qemu-kvm-common.x86_64 10:1.5.3-160.el7_6.2 \n\nComplete!\n"]} 2019-10-01T02:16:36.979 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=mod_fastcgi) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["mod_fastcgi"]}, "item": "mod_fastcgi", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:16:38.390 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["librados2"]}, "item": "librados2", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 5.5 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 1/1 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 \n\nComplete!\n"]} 2019-10-01T02:16:39.345 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 1.3 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:16:39.400 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:39 +0000 (0:00:23.147) 0:05:40.737 ******* 2019-10-01T02:16:40.954 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=mod_fastcgi) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["mod_fastcgi"]}, "item": "mod_fastcgi", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:16:41.975 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:16:41.977 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["boost-random"]}, "item": "boost-random", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-10-01T02:16:42.631 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=boost-program-options) => {"ansible_loop_var": "item", "changed": false, "item": "boost-program-options", "msg": "", "rc": 0, "results": ["boost-program-options is not installed"]} 2019-10-01T02:16:43.220 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": false, "item": "leveldb", "msg": "", "rc": 0, "results": ["leveldb is not installed"]} 2019-10-01T02:16:43.315 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-01T02:16:43.317 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 1.3 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:16:43.368 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:43 +0000 (0:00:03.968) 0:05:44.706 ******* 2019-10-01T02:16:43.838 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:16:43.839 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": false, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["xmlstarlet is not installed"]} 2019-10-01T02:16:44.595 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-jinja2) => {"ansible_loop_var": "item", "changed": false, "item": "python-jinja2", "msg": "", "rc": 0, "results": ["python-jinja2 is not installed"]} 2019-10-01T02:16:45.378 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-01T02:16:46.015 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": true, "changes": {"removed": ["boost-random"]}, "item": "boost-random", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-10-01T02:16:46.142 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-flask) => {"ansible_loop_var": "item", "changed": false, "item": "python-flask", "msg": "", "rc": 0, "results": ["python-flask is not installed"]} 2019-10-01T02:16:46.708 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=boost-program-options) => {"ansible_loop_var": "item", "changed": false, "item": "boost-program-options", "msg": "", "rc": 0, "results": ["boost-program-options is not installed"]} 2019-10-01T02:16:46.958 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-requests) => {"ansible_loop_var": "item", "changed": false, "item": "python-requests", "msg": "", "rc": 0, "results": ["python-requests is not installed"]} 2019-10-01T02:16:47.260 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=leveldb) => {"ansible_loop_var": "item", "changed": false, "item": "leveldb", "msg": "", "rc": 0, "results": ["leveldb is not installed"]} 2019-10-01T02:16:47.728 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": false, "item": "boost-random", "msg": "", "rc": 0, "results": ["boost-random is not installed"]} 2019-10-01T02:16:47.766 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=xmlstarlet) => {"ansible_loop_var": "item", "changed": false, "item": "xmlstarlet", "msg": "", "rc": 0, "results": ["xmlstarlet is not installed"]} 2019-10-01T02:16:48.481 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-urllib3) => {"ansible_loop_var": "item", "changed": false, "item": "python-urllib3", "msg": "", "rc": 0, "results": ["python-urllib3 is not installed"]} 2019-10-01T02:16:48.556 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-jinja2) => {"ansible_loop_var": "item", "changed": false, "item": "python-jinja2", "msg": "", "rc": 0, "results": ["python-jinja2 is not installed"]} 2019-10-01T02:16:49.266 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-babel) => {"ansible_loop_var": "item", "changed": false, "item": "python-babel", "msg": "", "rc": 0, "results": ["python-babel is not installed"]} 2019-10-01T02:16:49.351 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "msg": "", "rc": 0, "results": ["python-ceph is not installed"]} 2019-10-01T02:16:49.873 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=hdparm) => {"ansible_loop_var": "item", "changed": false, "item": "hdparm", "msg": "", "rc": 0, "results": ["hdparm is not installed"]} 2019-10-01T02:16:50.155 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-flask) => {"ansible_loop_var": "item", "changed": false, "item": "python-flask", "msg": "", "rc": 0, "results": ["python-flask is not installed"]} 2019-10-01T02:16:50.652 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-markupsafe) => {"ansible_loop_var": "item", "changed": false, "item": "python-markupsafe", "msg": "", "rc": 0, "results": ["python-markupsafe is not installed"]} 2019-10-01T02:16:50.936 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-requests) => {"ansible_loop_var": "item", "changed": false, "item": "python-requests", "msg": "", "rc": 0, "results": ["python-requests is not installed"]} 2019-10-01T02:16:51.435 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-werkzeug) => {"ansible_loop_var": "item", "changed": false, "item": "python-werkzeug", "msg": "", "rc": 0, "results": ["python-werkzeug is not installed"]} 2019-10-01T02:16:51.731 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=boost-random) => {"ansible_loop_var": "item", "changed": false, "item": "boost-random", "msg": "", "rc": 0, "results": ["boost-random is not installed"]} 2019-10-01T02:16:52.223 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-itsdangerous) => {"ansible_loop_var": "item", "changed": false, "item": "python-itsdangerous", "msg": "", "rc": 0, "results": ["python-itsdangerous is not installed"]} 2019-10-01T02:16:52.275 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:52 +0000 (0:00:08.906) 0:05:53.613 ******* 2019-10-01T02:16:52.507 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-urllib3) => {"ansible_loop_var": "item", "changed": false, "item": "python-urllib3", "msg": "", "rc": 0, "results": ["python-urllib3 is not installed"]} 2019-10-01T02:16:53.275 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-babel) => {"ansible_loop_var": "item", "changed": false, "item": "python-babel", "msg": "", "rc": 0, "results": ["python-babel is not installed"]} 2019-10-01T02:16:53.623 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:16:53.625 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=@core) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@core"]}, "item": "@core", "msg": "Warning: Group core does not have any packages to install.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nNothing to do\n"]} 2019-10-01T02:16:53.813 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:16:53.815 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=hdparm) => {"ansible_loop_var": "item", "changed": false, "item": "hdparm", "msg": "", "rc": 0, "results": ["hdparm is not installed"]} 2019-10-01T02:16:54.536 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-markupsafe) => {"ansible_loop_var": "item", "changed": false, "item": "python-markupsafe", "msg": "", "rc": 0, "results": ["python-markupsafe is not installed"]} 2019-10-01T02:16:54.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:16:54.991 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=@base) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@base"]}, "item": "@base", "msg": "Warning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nNothing to do\n"]} 2019-10-01T02:16:55.198 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:16:55.199 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-werkzeug) => {"ansible_loop_var": "item", "changed": false, "item": "python-werkzeug", "msg": "", "rc": 0, "results": ["python-werkzeug is not installed"]} 2019-10-01T02:16:55.615 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:16:55.617 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=yum-plugin-priorities) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-priorities", "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-50.el7.noarch providing yum-plugin-priorities is already installed"]} 2019-10-01T02:16:55.962 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:16:55.964 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-itsdangerous) => {"ansible_loop_var": "item", "changed": false, "item": "python-itsdangerous", "msg": "", "rc": 0, "results": ["python-itsdangerous is not installed"]} 2019-10-01T02:16:56.005 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:16:56 +0000 (0:00:03.729) 0:05:57.342 ******* 2019-10-01T02:16:56.189 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:16:56.191 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=yum-plugin-fastestmirror) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-fastestmirror", "msg": "", "rc": 0, "results": ["yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed"]} 2019-10-01T02:16:56.788 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=redhat-lsb) => {"ansible_loop_var": "item", "changed": false, "item": "redhat-lsb", "msg": "", "rc": 0, "results": ["redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed"]} 2019-10-01T02:16:57.350 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=@core) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@core"]}, "item": "@core", "msg": "Warning: Group core does not have any packages to install.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nNothing to do\n"]} 2019-10-01T02:16:57.498 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=sysstat) => {"ansible_loop_var": "item", "changed": false, "item": "sysstat", "msg": "", "rc": 0, "results": ["sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed"]} 2019-10-01T02:16:58.090 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=gdb) => {"ansible_loop_var": "item", "changed": false, "item": "gdb", "msg": "", "rc": 0, "results": ["gdb-7.6.1-114.el7.x86_64 providing gdb is already installed"]} 2019-10-01T02:16:58.682 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=git-all) => {"ansible_loop_var": "item", "changed": false, "item": "git-all", "msg": "", "rc": 0, "results": ["git-all-1.8.3.1-20.el7.noarch providing git-all is already installed"]} 2019-10-01T02:16:58.706 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=@base) => {"ansible_loop_var": "item", "changed": false, "changes": {"installed": ["@base"]}, "item": "@base", "msg": "Warning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nNothing to do\n"]} 2019-10-01T02:16:59.376 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-configobj) => {"ansible_loop_var": "item", "changed": false, "item": "python-configobj", "msg": "", "rc": 0, "results": ["python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed"]} 2019-10-01T02:16:59.378 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=yum-plugin-priorities) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-priorities", "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-50.el7.noarch providing yum-plugin-priorities is already installed"]} 2019-10-01T02:17:00.092 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=gcc-c++) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-c++", "msg": "", "rc": 0, "results": ["gcc-c++-4.8.5-36.el7_6.2.x86_64 providing gcc-c++ is already installed"]} 2019-10-01T02:17:00.105 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=yum-plugin-fastestmirror) => {"ansible_loop_var": "item", "changed": false, "item": "yum-plugin-fastestmirror", "msg": "", "rc": 0, "results": ["yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed"]} 2019-10-01T02:17:00.752 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libedit) => {"ansible_loop_var": "item", "changed": false, "item": "libedit", "msg": "", "rc": 0, "results": ["libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed"]} 2019-10-01T02:17:00.805 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=redhat-lsb) => {"ansible_loop_var": "item", "changed": false, "item": "redhat-lsb", "msg": "", "rc": 0, "results": ["redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed"]} 2019-10-01T02:17:01.460 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=openssl098e) => {"ansible_loop_var": "item", "changed": false, "item": "openssl098e", "msg": "", "rc": 0, "results": ["openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed"]} 2019-10-01T02:17:01.503 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=sysstat) => {"ansible_loop_var": "item", "changed": false, "item": "sysstat", "msg": "", "rc": 0, "results": ["sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed"]} 2019-10-01T02:17:01.504 INFO:teuthology.task.ansible.out: 2019-10-01T02:17:02.154 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=boost-thread) => {"ansible_loop_var": "item", "changed": false, "item": "boost-thread", "msg": "", "rc": 0, "results": ["boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed"]} 2019-10-01T02:17:02.197 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=gdb) => {"ansible_loop_var": "item", "changed": false, "item": "gdb", "msg": "", "rc": 0, "results": ["gdb-7.6.1-114.el7.x86_64 providing gdb is already installed"]} 2019-10-01T02:17:02.837 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=git-all) => {"ansible_loop_var": "item", "changed": false, "item": "git-all", "msg": "", "rc": 0, "results": ["git-all-1.8.3.1-20.el7.noarch providing git-all is already installed"]} 2019-10-01T02:17:02.860 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=xfsprogs) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs", "msg": "", "rc": 0, "results": ["xfsprogs-4.5.0-19.el7_6.x86_64 providing xfsprogs is already installed"]} 2019-10-01T02:17:03.533 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed"]} 2019-10-01T02:17:03.555 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-configobj) => {"ansible_loop_var": "item", "changed": false, "item": "python-configobj", "msg": "", "rc": 0, "results": ["python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed"]} 2019-10-01T02:17:04.255 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=parted) => {"ansible_loop_var": "item", "changed": false, "item": "parted", "msg": "", "rc": 0, "results": ["parted-3.1-29.el7.x86_64 providing parted is already installed"]} 2019-10-01T02:17:04.297 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=gcc-c++) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-c++", "msg": "", "rc": 0, "results": ["gcc-c++-4.8.5-36.el7_6.2.x86_64 providing gcc-c++ is already installed"]} 2019-10-01T02:17:04.921 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libgcrypt) => {"ansible_loop_var": "item", "changed": false, "item": "libgcrypt", "msg": "", "rc": 0, "results": ["libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed"]} 2019-10-01T02:17:04.953 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libedit) => {"ansible_loop_var": "item", "changed": false, "item": "libedit", "msg": "", "rc": 0, "results": ["libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed"]} 2019-10-01T02:17:05.523 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=fuse) => {"ansible_loop_var": "item", "changed": false, "item": "fuse", "msg": "", "rc": 0, "results": ["fuse-2.9.2-11.el7.x86_64 providing fuse is already installed"]} 2019-10-01T02:17:05.597 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=openssl098e) => {"ansible_loop_var": "item", "changed": false, "item": "openssl098e", "msg": "", "rc": 0, "results": ["openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed"]} 2019-10-01T02:17:06.226 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=fuse-libs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-libs", "msg": "", "rc": 0, "results": ["fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed"]} 2019-10-01T02:17:06.280 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=boost-thread) => {"ansible_loop_var": "item", "changed": false, "item": "boost-thread", "msg": "", "rc": 0, "results": ["boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed"]} 2019-10-01T02:17:06.895 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-01T02:17:06.982 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=xfsprogs) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs", "msg": "", "rc": 0, "results": ["xfsprogs-4.5.0-19.el7_6.x86_64 providing xfsprogs is already installed"]} 2019-10-01T02:17:07.622 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=openssl) => {"ansible_loop_var": "item", "changed": false, "item": "openssl", "msg": "", "rc": 0, "results": ["1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed"]} 2019-10-01T02:17:07.694 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed"]} 2019-10-01T02:17:08.312 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libuuid) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid", "msg": "", "rc": 0, "results": ["libuuid-2.23.2-59.el7_6.1.x86_64 providing libuuid is already installed"]} 2019-10-01T02:17:08.385 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=parted) => {"ansible_loop_var": "item", "changed": false, "item": "parted", "msg": "", "rc": 0, "results": ["parted-3.1-29.el7.x86_64 providing parted is already installed"]} 2019-10-01T02:17:08.989 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=btrfs-progs) => {"ansible_loop_var": "item", "changed": false, "item": "btrfs-progs", "msg": "", "rc": 0, "results": ["btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed"]} 2019-10-01T02:17:09.011 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libgcrypt) => {"ansible_loop_var": "item", "changed": false, "item": "libgcrypt", "msg": "", "rc": 0, "results": ["libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed"]} 2019-10-01T02:17:09.714 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=fuse) => {"ansible_loop_var": "item", "changed": false, "item": "fuse", "msg": "", "rc": 0, "results": ["fuse-2.9.2-11.el7.x86_64 providing fuse is already installed"]} 2019-10-01T02:17:09.715 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=attr) => {"ansible_loop_var": "item", "changed": false, "item": "attr", "msg": "", "rc": 0, "results": ["attr-2.4.46-13.el7.x86_64 providing attr is already installed"]} 2019-10-01T02:17:10.377 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=valgrind) => {"ansible_loop_var": "item", "changed": false, "item": "valgrind", "msg": "", "rc": 0, "results": ["1:valgrind-3.13.0-13.el7.x86_64 providing valgrind is already installed"]} 2019-10-01T02:17:10.423 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=fuse-libs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-libs", "msg": "", "rc": 0, "results": ["fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed"]} 2019-10-01T02:17:11.080 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-01T02:17:11.125 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-nose) => {"ansible_loop_var": "item", "changed": false, "item": "python-nose", "msg": "", "rc": 0, "results": ["python-nose-1.3.7-1.el7.noarch providing python-nose is already installed"]} 2019-10-01T02:17:11.823 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=openssl) => {"ansible_loop_var": "item", "changed": false, "item": "openssl", "msg": "", "rc": 0, "results": ["1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed"]} 2019-10-01T02:17:12.104 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "msg": "", "rc": 0, "results": ["mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed"]} 2019-10-01T02:17:12.523 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libuuid) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid", "msg": "", "rc": 0, "results": ["libuuid-2.23.2-59.el7_6.1.x86_64 providing libuuid is already installed"]} 2019-10-01T02:17:12.804 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ant) => {"ansible_loop_var": "item", "changed": false, "item": "ant", "msg": "", "rc": 0, "results": ["ant-1.9.4-2.el7.noarch providing ant is already installed"]} 2019-10-01T02:17:13.249 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=btrfs-progs) => {"ansible_loop_var": "item", "changed": false, "item": "btrfs-progs", "msg": "", "rc": 0, "results": ["btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed"]} 2019-10-01T02:17:13.964 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=attr) => {"ansible_loop_var": "item", "changed": false, "item": "attr", "msg": "", "rc": 0, "results": ["attr-2.4.46-13.el7.x86_64 providing attr is already installed"]} 2019-10-01T02:17:14.632 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=valgrind) => {"ansible_loop_var": "item", "changed": false, "item": "valgrind", "msg": "", "rc": 0, "results": ["1:valgrind-3.13.0-13.el7.x86_64 providing valgrind is already installed"]} 2019-10-01T02:17:15.357 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-nose) => {"ansible_loop_var": "item", "changed": false, "item": "python-nose", "msg": "", "rc": 0, "results": ["python-nose-1.3.7-1.el7.noarch providing python-nose is already installed"]} 2019-10-01T02:17:16.038 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 800 k\nInstalled size: 800 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:17:16.335 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "msg": "", "rc": 0, "results": ["mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed"]} 2019-10-01T02:17:16.655 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libtool) => {"ansible_loop_var": "item", "changed": false, "item": "libtool", "msg": "", "rc": 0, "results": ["libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed"]} 2019-10-01T02:17:17.042 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ant) => {"ansible_loop_var": "item", "changed": false, "item": "ant", "msg": "", "rc": 0, "results": ["ant-1.9.4-2.el7.noarch providing ant is already installed"]} 2019-10-01T02:17:17.353 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=automake) => {"ansible_loop_var": "item", "changed": false, "item": "automake", "msg": "", "rc": 0, "results": ["automake-1.13.4-3.el7.noarch providing automake is already installed"]} 2019-10-01T02:17:18.059 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=gettext) => {"ansible_loop_var": "item", "changed": false, "item": "gettext", "msg": "", "rc": 0, "results": ["gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed"]} 2019-10-01T02:17:18.795 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libuuid-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid-devel", "msg": "", "rc": 0, "results": ["libuuid-devel-2.23.2-59.el7_6.1.x86_64 providing libuuid-devel is already installed"]} 2019-10-01T02:17:19.590 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libacl-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libacl-devel", "msg": "", "rc": 0, "results": ["libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed"]} 2019-10-01T02:17:20.287 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=iozone) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["iozone"]}, "item": "iozone", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 800 k\nInstalled size: 800 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:17:20.330 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=bc) => {"ansible_loop_var": "item", "changed": false, "item": "bc", "msg": "", "rc": 0, "results": ["bc-1.06.95-13.el7.x86_64 providing bc is already installed"]} 2019-10-01T02:17:20.976 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libtool) => {"ansible_loop_var": "item", "changed": false, "item": "libtool", "msg": "", "rc": 0, "results": ["libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed"]} 2019-10-01T02:17:21.075 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=xfsdump) => {"ansible_loop_var": "item", "changed": false, "item": "xfsdump", "msg": "", "rc": 0, "results": ["xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed"]} 2019-10-01T02:17:21.708 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=automake) => {"ansible_loop_var": "item", "changed": false, "item": "automake", "msg": "", "rc": 0, "results": ["automake-1.13.4-3.el7.noarch providing automake is already installed"]} 2019-10-01T02:17:21.797 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=xfsprogs-devel) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs-devel", "msg": "", "rc": 0, "results": ["xfsprogs-devel-4.5.0-19.el7_6.x86_64 providing xfsprogs-devel is already installed"]} 2019-10-01T02:17:22.426 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=gettext) => {"ansible_loop_var": "item", "changed": false, "item": "gettext", "msg": "", "rc": 0, "results": ["gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed"]} 2019-10-01T02:17:22.523 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=blktrace) => {"ansible_loop_var": "item", "changed": false, "item": "blktrace", "msg": "", "rc": 0, "results": ["blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed"]} 2019-10-01T02:17:23.117 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libuuid-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libuuid-devel", "msg": "", "rc": 0, "results": ["libuuid-devel-2.23.2-59.el7_6.1.x86_64 providing libuuid-devel is already installed"]} 2019-10-01T02:17:23.248 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=numpy) => {"ansible_loop_var": "item", "changed": false, "item": "numpy", "msg": "", "rc": 0, "results": ["1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed"]} 2019-10-01T02:17:23.839 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libacl-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libacl-devel", "msg": "", "rc": 0, "results": ["libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed"]} 2019-10-01T02:17:23.982 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-matplotlib) => {"ansible_loop_var": "item", "changed": false, "item": "python-matplotlib", "msg": "", "rc": 0, "results": ["python-matplotlib-1.2.0-15.el7.x86_64 providing python-matplotlib is already installed"]} 2019-10-01T02:17:24.528 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=bc) => {"ansible_loop_var": "item", "changed": false, "item": "bc", "msg": "", "rc": 0, "results": ["bc-1.06.95-13.el7.x86_64 providing bc is already installed"]} 2019-10-01T02:17:25.217 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=xfsdump) => {"ansible_loop_var": "item", "changed": false, "item": "xfsdump", "msg": "", "rc": 0, "results": ["xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed"]} 2019-10-01T02:17:25.923 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=xfsprogs-devel) => {"ansible_loop_var": "item", "changed": false, "item": "xfsprogs-devel", "msg": "", "rc": 0, "results": ["xfsprogs-devel-4.5.0-19.el7_6.x86_64 providing xfsprogs-devel is already installed"]} 2019-10-01T02:17:26.615 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=blktrace) => {"ansible_loop_var": "item", "changed": false, "item": "blktrace", "msg": "", "rc": 0, "results": ["blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed"]} 2019-10-01T02:17:27.373 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=numpy) => {"ansible_loop_var": "item", "changed": false, "item": "numpy", "msg": "", "rc": 0, "results": ["1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed"]} 2019-10-01T02:17:28.099 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-matplotlib) => {"ansible_loop_var": "item", "changed": false, "item": "python-matplotlib", "msg": "", "rc": 0, "results": ["python-matplotlib-1.2.0-15.el7.x86_64 providing python-matplotlib is already installed"]} 2019-10-01T02:17:30.718 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=qemu-kvm) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["qemu-kvm"]}, "item": "qemu-kvm", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n l 2019-10-01T02:17:30.720 INFO:teuthology.task.ansible.out:ibrados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 8.0 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/6 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/6 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/6 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/6 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/6 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/6 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/6 \n\nInstalled:\n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n 2019-10-01T02:17:30.722 INFO:teuthology.task.ansible.out: librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-01T02:17:31.437 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=usbredir) => {"ansible_loop_var": "item", "changed": false, "item": "usbredir", "msg": "", "rc": 0, "results": ["usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed"]} 2019-10-01T02:17:32.124 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=genisoimage) => {"ansible_loop_var": "item", "changed": false, "item": "genisoimage", "msg": "", "rc": 0, "results": ["genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed"]} 2019-10-01T02:17:32.729 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=httpd) => {"ansible_loop_var": "item", "changed": false, "item": "httpd", "msg": "", "rc": 0, "results": ["httpd-2.4.6-89.el7.centos.x86_64 providing httpd is already installed"]} 2019-10-01T02:17:33.434 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=httpd-devel) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-devel", "msg": "", "rc": 0, "results": ["httpd-devel-2.4.6-89.el7.centos.x86_64 providing httpd-devel is already installed"]} 2019-10-01T02:17:34.088 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=httpd-tools) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-tools", "msg": "", "rc": 0, "results": ["httpd-tools-2.4.6-89.el7.centos.x86_64 providing httpd-tools is already installed"]} 2019-10-01T02:17:34.734 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=mod_ssl) => {"ansible_loop_var": "item", "changed": false, "item": "mod_ssl", "msg": "", "rc": 0, "results": ["1:mod_ssl-2.4.6-89.el7.centos.x86_64 providing mod_ssl is already installed"]} 2019-10-01T02:17:34.780 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=qemu-kvm) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["qemu-kvm"]}, "item": "qemu-kvm", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n 2019-10-01T02:17:34.782 INFO:teuthology.task.ansible.out: librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 7.9 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/6 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/6 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/6 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/6 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/6 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/6 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 3/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/6 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/6 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/6 \n\nInstalled:\n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \ 2019-10-01T02:17:34.784 INFO:teuthology.task.ansible.out:n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-10-01T02:17:35.549 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=usbredir) => {"ansible_loop_var": "item", "changed": false, "item": "usbredir", "msg": "", "rc": 0, "results": ["usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed"]} 2019-10-01T02:17:36.270 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=genisoimage) => {"ansible_loop_var": "item", "changed": false, "item": "genisoimage", "msg": "", "rc": 0, "results": ["genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed"]} 2019-10-01T02:17:36.953 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=httpd) => {"ansible_loop_var": "item", "changed": false, "item": "httpd", "msg": "", "rc": 0, "results": ["httpd-2.4.6-89.el7.centos.x86_64 providing httpd is already installed"]} 2019-10-01T02:17:37.621 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=httpd-devel) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-devel", "msg": "", "rc": 0, "results": ["httpd-devel-2.4.6-89.el7.centos.x86_64 providing httpd-devel is already installed"]} 2019-10-01T02:17:38.350 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=httpd-tools) => {"ansible_loop_var": "item", "changed": false, "item": "httpd-tools", "msg": "", "rc": 0, "results": ["httpd-tools-2.4.6-89.el7.centos.x86_64 providing httpd-tools is already installed"]} 2019-10-01T02:17:39.059 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=mod_ssl) => {"ansible_loop_var": "item", "changed": false, "item": "mod_ssl", "msg": "", "rc": 0, "results": ["1:mod_ssl-2.4.6-89.el7.centos.x86_64 providing mod_ssl is already installed"]} 2019-10-01T02:17:52.726 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item=mod_fastcgi-2.4.7-1.ceph.el7.centos) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "item": "mod_fastcgi-2.4.7-1.ceph.el7.centos", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 70 k\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:17:53.412 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=libev-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libev-devel", "msg": "", "rc": 0, "results": ["libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed"]} 2019-10-01T02:17:54.135 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=perl-XML-Twig) => {"ansible_loop_var": "item", "changed": false, "item": "perl-XML-Twig", "msg": "", "rc": 0, "results": ["perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed"]} 2019-10-01T02:17:54.803 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=java-1.6.0-openjdk-devel) => {"ansible_loop_var": "item", "changed": false, "item": "java-1.6.0-openjdk-devel", "msg": "", "rc": 0, "results": ["1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed"]} 2019-10-01T02:17:55.873 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=junit4) => {"ansible_loop_var": "item", "changed": false, "item": "junit4", "msg": "", "rc": 0, "results": ["junit-4.11-8.el7.noarch providing junit4 is already installed"]} 2019-10-01T02:17:56.594 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=nfs-utils) => {"ansible_loop_var": "item", "changed": false, "item": "nfs-utils", "msg": "", "rc": 0, "results": ["1:nfs-utils-1.3.0-0.35.el7.x86_64 providing nfs-utils is already installed"]} 2019-10-01T02:17:57.060 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item=mod_fastcgi-2.4.7-1.ceph.el7.centos) => {"ansible_loop_var": "item", "changed": true, "changes": {"installed": ["mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "item": "mod_fastcgi-2.4.7-1.ceph.el7.centos", "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: repo1.ash.innoscale.net\n * updates: mirror.jaleco.com\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 70 k\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nComplete!\n"]} 2019-10-01T02:17:57.315 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=ncurses-devel) => {"ansible_loop_var": "item", "changed": false, "item": "ncurses-devel", "msg": "", "rc": 0, "results": ["ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed"]} 2019-10-01T02:17:57.806 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=libev-devel) => {"ansible_loop_var": "item", "changed": false, "item": "libev-devel", "msg": "", "rc": 0, "results": ["libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed"]} 2019-10-01T02:17:58.018 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-devel) => {"ansible_loop_var": "item", "changed": false, "item": "python-devel", "msg": "", "rc": 0, "results": ["python-devel-2.7.5-80.el7_6.x86_64 providing python-devel is already installed"]} 2019-10-01T02:17:58.502 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=perl-XML-Twig) => {"ansible_loop_var": "item", "changed": false, "item": "perl-XML-Twig", "msg": "", "rc": 0, "results": ["perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed"]} 2019-10-01T02:17:58.737 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-01T02:17:59.141 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=java-1.6.0-openjdk-devel) => {"ansible_loop_var": "item", "changed": false, "item": "java-1.6.0-openjdk-devel", "msg": "", "rc": 0, "results": ["1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed"]} 2019-10-01T02:17:59.437 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=perl-CPAN) => {"ansible_loop_var": "item", "changed": false, "item": "perl-CPAN", "msg": "", "rc": 0, "results": ["perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed"]} 2019-10-01T02:17:59.537 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:17:59 +0000 (0:01:03.532) 0:07:00.874 ******* 2019-10-01T02:18:00.078 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=junit4) => {"ansible_loop_var": "item", "changed": false, "item": "junit4", "msg": "", "rc": 0, "results": ["junit-4.11-8.el7.noarch providing junit4 is already installed"]} 2019-10-01T02:18:00.352 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:00.353 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=cryptopp-devel) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp-devel", "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed"]} 2019-10-01T02:18:00.806 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:00.808 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=nfs-utils) => {"ansible_loop_var": "item", "changed": false, "item": "nfs-utils", "msg": "", "rc": 0, "results": ["1:nfs-utils-1.3.0-0.35.el7.x86_64 providing nfs-utils is already installed"]} 2019-10-01T02:18:01.144 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:01.146 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=cryptopp) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp", "msg": "", "rc": 0, "results": ["cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed"]} 2019-10-01T02:18:01.519 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:01.520 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=ncurses-devel) => {"ansible_loop_var": "item", "changed": false, "item": "ncurses-devel", "msg": "", "rc": 0, "results": ["ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed"]} 2019-10-01T02:18:01.884 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:01.886 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=fcgi) => {"ansible_loop_var": "item", "changed": false, "item": "fcgi", "msg": "", "rc": 0, "results": ["fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed"]} 2019-10-01T02:18:02.228 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:02.229 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=python-devel) => {"ansible_loop_var": "item", "changed": false, "item": "python-devel", "msg": "", "rc": 0, "results": ["python-devel-2.7.5-80.el7_6.x86_64 providing python-devel is already installed"]} 2019-10-01T02:18:02.659 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:02.660 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=dbench) => {"ansible_loop_var": "item", "changed": false, "item": "dbench", "msg": "", "rc": 0, "results": ["dbench-4.0-10.el7.x86_64 providing dbench is already installed"]} 2019-10-01T02:18:02.909 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* ok: [mira064.front.sepia.ceph.com] => (item=python-virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "python-virtualenv", "msg": "", "rc": 0, "results": ["python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed"]} 2019-10-01T02:18:03.451 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [mira063.front.sepia.ceph.com] => (item=fuse-sshfs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-sshfs", "msg": "", "rc": 0, "results": ["fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed"]} 2019-10-01T02:18:03.638 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* ok: [mira064.front.sepia.ceph.com] => (item=perl-CPAN) => {"ansible_loop_var": "item", "changed": false, "item": "perl-CPAN", "msg": "", "rc": 0, "results": ["perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed"]} 2019-10-01T02:18:03.729 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:03 +0000 (0:00:04.191) 0:07:05.066 ******* 2019-10-01T02:18:04.194 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [mira063.front.sepia.ceph.com] => (item=bonnie++) => {"ansible_loop_var": "item", "changed": false, "item": "bonnie++", "msg": "", "rc": 0, "results": ["bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed"]} 2019-10-01T02:18:04.508 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=cryptopp-devel) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp-devel", "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed"]} 2019-10-01T02:18:04.905 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=perl-JSON-XS) => {"ansible_loop_var": "item", "changed": false, "item": "perl-JSON-XS", "msg": "", "rc": 0, "results": ["1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-10-01T02:18:04.942 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:04 +0000 (0:00:01.213) 0:07:06.280 ******* 2019-10-01T02:18:04.972 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-01T02:18:05.017 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:05 +0000 (0:00:00.074) 0:07:06.354 ******* 2019-10-01T02:18:05.048 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:05.083 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:05 +0000 (0:00:00.066) 0:07:06.421 ******* 2019-10-01T02:18:05.330 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:05.331 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=cryptopp) => {"ansible_loop_var": "item", "changed": false, "item": "cryptopp", "msg": "", "rc": 0, "results": ["cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed"]} 2019-10-01T02:18:05.677 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ipset.service shutdown.target ebtables.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "ye 2019-10-01T02:18:05.680 INFO:teuthology.task.ansible.out:s", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriori 2019-10-01T02:18:05.681 INFO:teuthology.task.ansible.out:ty": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-01T02:18:05.715 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:05 +0000 (0:00:00.631) 0:07:07.053 ******* 2019-10-01T02:18:06.115 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:06.115 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=fcgi) => {"ansible_loop_var": "item", "changed": false, "item": "fcgi", "msg": "", "rc": 0, "results": ["fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed"]} 2019-10-01T02:18:06.280 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [mira063.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-10-01T02:18:06.316 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:06 +0000 (0:00:00.598) 0:07:07.651 ******* 2019-10-01T02:18:06.857 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:06.858 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=dbench) => {"ansible_loop_var": "item", "changed": false, "item": "dbench", "msg": "", "rc": 0, "results": ["dbench-4.0-10.el7.x86_64 providing dbench is already installed"]} 2019-10-01T02:18:07.323 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-10-01T02:18:07.357 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:01.042) 0:07:08.694 ******* 2019-10-01T02:18:07.391 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-01T02:18:07.393 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.437 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.079) 0:07:08.774 ******* 2019-10-01T02:18:07.468 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-01T02:18:07.515 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.078) 0:07:08.852 ******* 2019-10-01T02:18:07.544 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-01T02:18:07.545 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.581 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.066) 0:07:08.918 ******* 2019-10-01T02:18:07.613 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:07.617 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=fuse-sshfs) => {"ansible_loop_var": "item", "changed": false, "item": "fuse-sshfs", "msg": "", "rc": 0, "results": ["fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed"]} TASK [testnode : Add sources list] ********************************************* skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.656 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.075) 0:07:08.993 ******* 2019-10-01T02:18:07.694 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-01T02:18:07.696 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.706 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.755 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.098) 0:07:09.092 ******* 2019-10-01T02:18:07.785 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-01T02:18:07.787 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.819 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.064) 0:07:09.157 ******* 2019-10-01T02:18:07.848 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-01T02:18:07.898 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.078) 0:07:09.236 ******* 2019-10-01T02:18:07.930 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-01T02:18:07.931 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:07.979 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:07 +0000 (0:00:00.080) 0:07:09.316 ******* 2019-10-01T02:18:08.005 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-01T02:18:08.007 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.040 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.061) 0:07:09.377 ******* 2019-10-01T02:18:08.068 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:08.142 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:08.144 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-01T02:18:08.145 INFO:teuthology.task.ansible.out: skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.189 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.149) 0:07:09.527 ******* 2019-10-01T02:18:08.218 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:08.257 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.067) 0:07:09.595 ******* 2019-10-01T02:18:08.291 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-01T02:18:08.343 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.085) 0:07:09.680 ******* 2019-10-01T02:18:08.359 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:08.360 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=bonnie++) => {"ansible_loop_var": "item", "changed": false, "item": "bonnie++", "msg": "", "rc": 0, "results": ["bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed"]} 2019-10-01T02:18:08.375 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:08.554 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:08.555 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.622 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.278) 0:07:09.959 ******* 2019-10-01T02:18:08.652 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-01T02:18:08.689 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.067) 0:07:10.026 ******* 2019-10-01T02:18:08.727 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-01T02:18:08.729 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.770 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.080) 0:07:10.107 ******* 2019-10-01T02:18:08.829 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-01T02:18:08.831 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.868 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.098) 0:07:10.206 ******* 2019-10-01T02:18:08.908 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-01T02:18:08.910 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:08.950 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:08 +0000 (0:00:00.081) 0:07:10.287 ******* 2019-10-01T02:18:08.978 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-01T02:18:08.979 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.029 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.078) 0:07:10.365 ******* 2019-10-01T02:18:09.055 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-01T02:18:09.057 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.105 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.076) 0:07:10.442 ******* 2019-10-01T02:18:09.112 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-01T02:18:09.114 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=perl-JSON-XS) => {"ansible_loop_var": "item", "changed": false, "item": "perl-JSON-XS", "msg": "", "rc": 0, "results": ["1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-10-01T02:18:09.169 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.063) 0:07:10.506 ******* 2019-10-01T02:18:09.175 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-01T02:18:09.176 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.217 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.048) 0:07:10.555 ******* 2019-10-01T02:18:09.224 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-01T02:18:09.270 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.052) 0:07:10.607 ******* 2019-10-01T02:18:09.278 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-01T02:18:09.279 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.339 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.068) 0:07:10.675 ******* 2019-10-01T02:18:09.344 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:09.386 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.045) 0:07:10.721 ******* 2019-10-01T02:18:09.389 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-01T02:18:09.392 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.429 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.045) 0:07:10.766 ******* 2019-10-01T02:18:09.468 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-01T02:18:09.469 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.509 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.080) 0:07:10.846 ******* 2019-10-01T02:18:09.537 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-01T02:18:09.539 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.583 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.073) 0:07:10.920 ******* 2019-10-01T02:18:09.621 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.655 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.071) 0:07:10.992 ******* 2019-10-01T02:18:09.681 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-01T02:18:09.683 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.720 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.064) 0:07:11.057 ******* 2019-10-01T02:18:09.748 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.799 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.079) 0:07:11.136 ******* 2019-10-01T02:18:09.824 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:09.892 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-01T02:18:09.894 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:09.947 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:09 +0000 (0:00:00.146) 0:07:11.282 ******* 2019-10-01T02:18:09.984 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_dependency_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:09.991 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-01T02:18:09.993 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.service polkit.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ipset.service ip6tables.service ebtables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Ignore 2019-10-01T02:18:09.994 INFO:teuthology.task.ansible.out:OnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYV 2019-10-01T02:18:09.997 INFO:teuthology.task.ansible.out:TDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-01T02:18:10.027 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.082) 0:07:11.365 ******* 2019-10-01T02:18:10.055 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:18:10.057 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:10.106 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.078) 0:07:11.443 ******* 2019-10-01T02:18:10.138 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-01T02:18:10.180 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.072) 0:07:11.516 ******* 2019-10-01T02:18:10.212 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:10.395 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:10.397 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:10.448 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.268) 0:07:11.785 ******* 2019-10-01T02:18:10.476 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:10.516 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.063) 0:07:11.848 ******* 2019-10-01T02:18:10.550 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-01T02:18:10.552 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:10.626 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.114) 0:07:11.963 ******* 2019-10-01T02:18:10.645 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-01T02:18:10.647 INFO:teuthology.task.ansible.out:fatal: [mira064.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-10-01T02:18:10.683 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:10 +0000 (0:00:00.056) 0:07:12.020 ******* 2019-10-01T02:18:11.104 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-10-01T02:18:11.145 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.462) 0:07:12.482 ******* 2019-10-01T02:18:11.265 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-10-01T02:18:11.267 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-10-01T02:18:11.324 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.178) 0:07:12.661 ******* 2019-10-01T02:18:11.365 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-01T02:18:11.367 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.408 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.083) 0:07:12.745 ******* 2019-10-01T02:18:11.444 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-01T02:18:11.485 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.077) 0:07:12.822 ******* 2019-10-01T02:18:11.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-01T02:18:11.516 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.564 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.078) 0:07:12.901 ******* 2019-10-01T02:18:11.596 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.631 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.067) 0:07:12.968 ******* 2019-10-01T02:18:11.638 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-01T02:18:11.640 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-01T02:18:11.676 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.044) 0:07:13.013 ******* 2019-10-01T02:18:11.682 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-01T02:18:11.683 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.685 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.738 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.061) 0:07:13.075 ******* 2019-10-01T02:18:11.770 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-01T02:18:11.771 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.828 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.084) 0:07:13.159 ******* 2019-10-01T02:18:11.852 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-01T02:18:11.891 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.069) 0:07:13.229 ******* 2019-10-01T02:18:11.919 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:11.958 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:11 +0000 (0:00:00.066) 0:07:13.295 ******* 2019-10-01T02:18:11.999 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-01T02:18:12.001 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.041 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.082) 0:07:13.378 ******* 2019-10-01T02:18:12.072 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:12.112 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. 2019-10-01T02:18:12.114 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-10-01T02:18:12.161 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.118) 0:07:13.496 ******* 2019-10-01T02:18:12.167 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-01T02:18:12.168 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.223 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.063) 0:07:13.560 ******* 2019-10-01T02:18:12.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.282 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.059) 0:07:13.620 ******* 2019-10-01T02:18:12.291 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:12.347 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.064) 0:07:13.684 ******* 2019-10-01T02:18:12.356 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-01T02:18:12.359 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.401 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.053) 0:07:13.738 ******* 2019-10-01T02:18:12.407 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-01T02:18:12.457 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.055) 0:07:13.794 ******* 2019-10-01T02:18:12.465 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-01T02:18:12.467 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.482 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:12.514 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.056) 0:07:13.851 ******* 2019-10-01T02:18:12.542 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-01T02:18:12.543 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.581 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.063) 0:07:13.914 ******* 2019-10-01T02:18:12.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.648 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.070) 0:07:13.985 ******* 2019-10-01T02:18:12.656 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-01T02:18:12.657 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.702 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.053) 0:07:14.039 ******* 2019-10-01T02:18:12.709 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-01T02:18:12.710 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.757 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.055) 0:07:14.095 ******* 2019-10-01T02:18:12.765 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-01T02:18:12.808 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.050) 0:07:14.146 ******* 2019-10-01T02:18:12.815 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.860 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.051) 0:07:14.197 ******* 2019-10-01T02:18:12.866 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-01T02:18:12.868 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.906 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.046) 0:07:14.243 ******* 2019-10-01T02:18:12.910 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-01T02:18:12.913 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:12.949 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:12 +0000 (0:00:00.043) 0:07:14.287 ******* 2019-10-01T02:18:12.951 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:12.955 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-01T02:18:12.957 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.009 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.059) 0:07:14.346 ******* 2019-10-01T02:18:13.016 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-01T02:18:13.017 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.055 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.046) 0:07:14.392 ******* 2019-10-01T02:18:13.061 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-01T02:18:13.063 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.098 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.042) 0:07:14.435 ******* 2019-10-01T02:18:13.103 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-01T02:18:13.103 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.144 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.045) 0:07:14.481 ******* 2019-10-01T02:18:13.161 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-01T02:18:13.162 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.198 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.054) 0:07:14.535 ******* 2019-10-01T02:18:13.203 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-01T02:18:13.205 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.253 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.055) 0:07:14.590 ******* 2019-10-01T02:18:13.255 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:13.261 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.306 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.053) 0:07:14.643 ******* 2019-10-01T02:18:13.311 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-01T02:18:13.312 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.350 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:14.687 ******* 2019-10-01T02:18:13.356 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-01T02:18:13.358 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.392 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.042) 0:07:14.729 ******* 2019-10-01T02:18:13.400 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-01T02:18:13.402 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.436 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:14.773 ******* 2019-10-01T02:18:13.444 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-01T02:18:13.447 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.490 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.053) 0:07:14.827 ******* 2019-10-01T02:18:13.504 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.549 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.059) 0:07:14.886 ******* 2019-10-01T02:18:13.555 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-01T02:18:13.557 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.603 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.054) 0:07:14.940 ******* 2019-10-01T02:18:13.608 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.648 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.045) 0:07:14.985 ******* 2019-10-01T02:18:13.657 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-01T02:18:13.659 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.696 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.047) 0:07:15.033 ******* 2019-10-01T02:18:13.703 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-01T02:18:13.705 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.739 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:15.077 ******* 2019-10-01T02:18:13.745 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-01T02:18:13.747 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.787 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.047) 0:07:15.124 ******* 2019-10-01T02:18:13.793 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-01T02:18:13.794 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.831 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:15.168 ******* 2019-10-01T02:18:13.836 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-01T02:18:13.838 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.875 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:15.212 ******* 2019-10-01T02:18:13.882 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-01T02:18:13.884 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.918 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.043) 0:07:15.255 ******* 2019-10-01T02:18:13.925 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-01T02:18:13.927 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:13.967 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:13 +0000 (0:00:00.047) 0:07:15.303 ******* 2019-10-01T02:18:13.972 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-01T02:18:13.974 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.016 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.047) 0:07:15.351 ******* 2019-10-01T02:18:14.020 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-01T02:18:14.021 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.062 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.048) 0:07:15.399 ******* 2019-10-01T02:18:14.067 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-01T02:18:14.069 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.083 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:14.106 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.043) 0:07:15.443 ******* 2019-10-01T02:18:14.137 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.169 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.063) 0:07:15.507 ******* 2019-10-01T02:18:14.175 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-01T02:18:14.176 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.218 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.047) 0:07:15.554 ******* 2019-10-01T02:18:14.222 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-01T02:18:14.224 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.243 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_dependency_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:14.264 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.047) 0:07:15.601 ******* 2019-10-01T02:18:14.292 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:18:14.294 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.332 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.067) 0:07:15.669 ******* 2019-10-01T02:18:14.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-01T02:18:14.340 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.387 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.054) 0:07:15.724 ******* 2019-10-01T02:18:14.394 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-01T02:18:14.394 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sdd) => {"ansible_loop_var": "item", "changed": false, "item": "sdd", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.396 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sde) => {"ansible_loop_var": "item", "changed": false, "item": "sde", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.409 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sdf) => {"ansible_loop_var": "item", "changed": false, "item": "sdf", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.409 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sdg) => {"ansible_loop_var": "item", "changed": false, "item": "sdg", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.411 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item=sdb) => {"ansible_loop_var": "item", "changed": false, "item": "sdb", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.422 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=sdc) => {"ansible_loop_var": "item", "changed": false, "item": "sdc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.423 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-01T02:18:14.424 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [mira063.front.sepia.ceph.com] => (item=sdh) => {"ansible_loop_var": "item", "changed": false, "item": "sdh", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.426 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.467 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.078) 0:07:15.802 ******* 2019-10-01T02:18:14.495 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.030) 0:07:15.833 ******* 2019-10-01T02:18:14.501 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-01T02:18:14.505 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.515 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:14.542 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.046) 0:07:15.879 ******* 2019-10-01T02:18:14.569 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-01T02:18:14.570 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.607 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.063) 0:07:15.943 ******* 2019-10-01T02:18:14.643 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-01T02:18:14.644 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.694 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.088) 0:07:16.031 ******* 2019-10-01T02:18:14.700 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.757 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.061) 0:07:16.093 ******* 2019-10-01T02:18:14.763 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.809 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.053) 0:07:16.147 ******* 2019-10-01T02:18:14.820 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-01T02:18:14.829 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:14.861 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.051) 0:07:16.198 ******* 2019-10-01T02:18:14.872 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [mira063.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.909 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.047) 0:07:16.246 ******* 2019-10-01T02:18:14.915 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-01T02:18:14.916 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:14.952 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:14 +0000 (0:00:00.043) 0:07:16.290 ******* 2019-10-01T02:18:15.412 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-01T02:18:15.413 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-10-01T02:18:15.453 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:15 +0000 (0:00:00.499) 0:07:16.789 ******* 2019-10-01T02:18:15.622 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [mira063.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed"]} 2019-10-01T02:18:15.886 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-01T02:18:15.887 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-10-01T02:18:15.941 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:15 +0000 (0:00:00.488) 0:07:17.278 ******* 2019-10-01T02:18:16.323 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [mira063.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "msg": "", "rc": 0, "results": ["7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-10-01T02:18:16.362 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.419) 0:07:17.698 ******* 2019-10-01T02:18:16.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** ok: [mira064.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-10-01T02:18:16.420 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.059) 0:07:17.757 ******* 2019-10-01T02:18:16.429 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-01T02:18:16.431 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item={u'block_used': 12996204, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328280, u'size_available': 931136634880, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328280, "block_size": 4096, "block_total": 240324484, "block_used": 12996204, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136634880, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}} 2019-10-01T02:18:16.486 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.065) 0:07:17.823 ******* 2019-10-01T02:18:16.495 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-01T02:18:16.496 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.547 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.061) 0:07:17.884 ******* 2019-10-01T02:18:16.588 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-01T02:18:16.590 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.649 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.102) 0:07:17.987 ******* 2019-10-01T02:18:16.678 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-01T02:18:16.679 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.716 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.066) 0:07:18.053 ******* 2019-10-01T02:18:16.757 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.790 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.073) 0:07:18.127 ******* 2019-10-01T02:18:16.826 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-01T02:18:16.828 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.879 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.087) 0:07:18.214 ******* 2019-10-01T02:18:16.916 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:16.954 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.076) 0:07:18.291 ******* 2019-10-01T02:18:16.959 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-01T02:18:16.960 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.014904", "end": "2019-10-01 02:18:16.921535", "rc": 0, "start": "2019-10-01 02:18:16.906631", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:17.001 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:16 +0000 (0:00:00.046) 0:07:18.338 ******* 2019-10-01T02:18:17.008 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-01T02:18:17.010 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.059 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.058) 0:07:18.396 ******* 2019-10-01T02:18:17.066 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-01T02:18:17.067 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'block_used': 12996204, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328280, u'size_available': 931136634880, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328280, "block_size": 4096, "block_total": 240324484, "block_used": 12996204, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136634880, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.116 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.056) 0:07:18.453 ******* 2019-10-01T02:18:17.123 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-01T02:18:17.124 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.162 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.046) 0:07:18.500 ******* 2019-10-01T02:18:17.170 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-01T02:18:17.210 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.045) 0:07:18.546 ******* 2019-10-01T02:18:17.216 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.251 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.042) 0:07:18.589 ******* 2019-10-01T02:18:17.283 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.318 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.066) 0:07:18.655 ******* 2019-10-01T02:18:17.345 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.380 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.062) 0:07:18.718 ******* 2019-10-01T02:18:17.423 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.471 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.090) 0:07:18.808 ******* 2019-10-01T02:18:17.501 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-01T02:18:17.503 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.542 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.070) 0:07:18.879 ******* 2019-10-01T02:18:17.580 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-01T02:18:17.582 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.614 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.072) 0:07:18.951 ******* 2019-10-01T02:18:17.652 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.700 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.085) 0:07:19.037 ******* 2019-10-01T02:18:17.727 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-01T02:18:17.729 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.793 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.086) 0:07:19.123 ******* 2019-10-01T02:18:17.826 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-01T02:18:17.828 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.861 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.075) 0:07:19.199 ******* 2019-10-01T02:18:17.888 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-01T02:18:17.890 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.923 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.060) 0:07:19.260 ******* 2019-10-01T02:18:17.951 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:17.997 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:17 +0000 (0:00:00.074) 0:07:19.334 ******* 2019-10-01T02:18:18.027 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.073 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.072) 0:07:19.407 ******* 2019-10-01T02:18:18.096 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-01T02:18:18.098 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.132 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.061) 0:07:19.469 ******* 2019-10-01T02:18:18.169 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-01T02:18:18.171 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.207 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.075) 0:07:19.544 ******* 2019-10-01T02:18:18.234 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.273 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.064) 0:07:19.609 ******* 2019-10-01T02:18:18.325 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [mira064.front.sepia.ceph.com] => (item=sdd) => {"ansible_loop_var": "item", "changed": false, "item": "sdd", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.336 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sde) => {"ansible_loop_var": "item", "changed": false, "item": "sde", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.337 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sdf) => {"ansible_loop_var": "item", "changed": false, "item": "sdf", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.338 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sdg) => {"ansible_loop_var": "item", "changed": false, "item": "sdg", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.350 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.350 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sdb) => {"ansible_loop_var": "item", "changed": false, "item": "sdb", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.353 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sdc) => {"ansible_loop_var": "item", "changed": false, "item": "sdc", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.369 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=sdh) => {"ansible_loop_var": "item", "changed": false, "item": "sdh", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.412 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.140) 0:07:19.749 ******* 2019-10-01T02:18:18.449 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-01T02:18:18.451 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.492 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.078) 0:07:19.828 ******* 2019-10-01T02:18:18.520 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.558 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.067) 0:07:19.896 ******* 2019-10-01T02:18:18.585 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-01T02:18:18.587 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.632 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.073) 0:07:19.969 ******* 2019-10-01T02:18:18.659 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-10-01T02:18:18.660 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.699 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.065) 0:07:20.035 ******* 2019-10-01T02:18:18.724 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:18.757 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-01T02:18:18.759 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-01T02:18:18.798 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:18 +0000 (0:00:00.100) 0:07:20.135 ******* 2019-10-01T02:18:18.804 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-01T02:18:18.806 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010070802'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdd'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdd || sgdisk --zap-all /dev/sdd", "delta": "0:00:01.043556", "end": "2019-10-01 02:18:18.726000", "item": {"key": "sdd", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010070802"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:17.682444", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:19.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-01T02:18:19.512 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed"]} 2019-10-01T02:18:20.153 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "msg": "", "rc": 0, "results": ["7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-10-01T02:18:20.197 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:20 +0000 (0:00:01.398) 0:07:21.534 ******* 2019-10-01T02:18:20.270 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [mira064.front.sepia.ceph.com] => (item={u'block_used': 12996189, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328295, u'size_available': 931136696320, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328295, "block_size": 4096, "block_total": 240324484, "block_used": 12996189, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136696320, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}} 2019-10-01T02:18:20.309 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:20 +0000 (0:00:00.112) 0:07:21.646 ******* 2019-10-01T02:18:20.327 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-01T02:18:20.330 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2070345100'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N20GDSAE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sde'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sde || sgdisk --zap-all /dev/sde", "delta": "0:00:01.020461", "end": "2019-10-01 02:18:20.227014", "item": {"key": "sde", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2070345100"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N20GDSAE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:19.206553", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:20.731 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.016738", "end": "2019-10-01 02:18:20.692598", "rc": 0, "start": "2019-10-01 02:18:20.675860", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:20.789 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:20 +0000 (0:00:00.479) 0:07:22.126 ******* 2019-10-01T02:18:20.840 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-01T02:18:20.841 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'block_used': 12996189, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328295, u'size_available': 931136696320, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328295, "block_size": 4096, "block_total": 240324484, "block_used": 12996189, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136696320, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:20.876 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:20 +0000 (0:00:00.087) 0:07:22.214 ******* 2019-10-01T02:18:20.917 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-01T02:18:20.970 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:20 +0000 (0:00:00.093) 0:07:22.307 ******* 2019-10-01T02:18:21.796 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Seagate', u'links': {u'masters': [u'dm-0'], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'6VPC4GD1', u'holders': [u'mpatha'], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST31000528AS', u'partitions': {}}, 'key': u'sdf'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdf || sgdisk --zap-all /dev/sdf", "delta": "0:00:01.019880", "end": "2019-10-01 02:18:21.742071", "item": {"key": "sdf", "value": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": [], "labels": [], "masters": ["dm-0"], "uuids": []}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:20.722191", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:22.513 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001755c00'], u'uuids': [u'08894f69-ea05-45a2-a056-85d846cce4c6']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N210EWLE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdd'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdd || sgdisk --zap-all /dev/sdd", "delta": "0:00:01.036168", "end": "2019-10-01 02:18:22.467947", "item": {"key": "sdd", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001755c00"], "labels": [], "masters": [], "uuids": ["08894f69-ea05-45a2-a056-85d846cce4c6"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210EWLE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:21.431779", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:23.288 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010866207'], u'uuids': [u'a560672c-38f4-4d12-9eb7-f21f6fd80dba']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdg'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdg || sgdisk --zap-all /dev/sdg", "delta": "0:00:01.020467", "end": "2019-10-01 02:18:23.242535", "item": {"key": "sdg", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010866207"], "labels": [], "masters": [], "uuids": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"]}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:22.222068", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:23.334 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010960009'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010960009-part1'], u'uuids': [u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010960009"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d2010960009-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:24.002 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20a0443200'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N20ZTT2C', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sde'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sde || sgdisk --zap-all /dev/sde", "delta": "0:00:01.020155", "end": "2019-10-01 02:18:23.950665", "item": {"key": "sde", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20a0443200"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZTT2C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:22.930510", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:24.803 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010572303'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdb'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdb || sgdisk --zap-all /dev/sdb", "delta": "0:00:01.018693", "end": "2019-10-01 02:18:24.752894", "item": {"key": "sdb", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010572303"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:23.734201", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:25.474 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001373100'], u'uuids': [u'ac2b8ca6-941f-4b45-90a5-6efdddf15630']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N21073AC', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdf'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdf || sgdisk --zap-all /dev/sdf", "delta": "0:00:01.018946", "end": "2019-10-01 02:18:25.431757", "item": {"key": "sdf", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001373100"], "labels": [], "masters": [], "uuids": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N21073AC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:24.412811", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:26.326 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010865804'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdc'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdc || sgdisk --zap-all /dev/sdc", "delta": "0:00:01.058371", "end": "2019-10-01 02:18:26.279072", "item": {"key": "sdc", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010865804"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:25.220701", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:26.935 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001475400'], u'uuids': [u'adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N210WTDE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdg'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdg || sgdisk --zap-all /dev/sdg", "delta": "0:00:01.019440", "end": "2019-10-01 02:18:26.893112", "item": {"key": "sdg", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001475400"], "labels": [], "masters": [], "uuids": ["adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210WTDE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:25.873672", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:26.950 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20114a3400'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N211JT4C', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20114a3400-part1'], u'uuids': [u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20114a3400"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d20114a3400-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211JT4C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:27.821 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Seagate', u'links': {u'masters': [u'dm-0'], u'labels': [], u'ids': [u'scsi-2001b4d2000000000'], u'uuids': [u'11dd6939-301d-45ff-b781-d2460ec22316']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'9VP9NVCL', u'holders': [u'mpatha'], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST31000528AS', u'partitions': {}}, 'key': u'sdh'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdh || sgdisk --zap-all /dev/sdh", "delta": "0:00:01.050879", "end": "2019-10-01 02:18:27.779831", "item": {"key": "sdh", "value": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2000000000"], "labels": [], "masters": ["dm-0"], "uuids": ["11dd6939-301d-45ff-b781-d2460ec22316"]}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "9VP9NVCL", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:26.728952", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:28.416 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20115e3600'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N211NUVC', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdb'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdb || sgdisk --zap-all /dev/sdb", "delta": "0:00:01.019116", "end": "2019-10-01 02:18:28.352273", "item": {"key": "sdb", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20115e3600"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211NUVC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:27.333157", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:29.320 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-mpatha', u'dm-uuid-mpath-2001b4d2000000000'], u'uuids': [u'7e986d2e-887e-4bc5-b72b-2c2e4214f494']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'serial': u'6VPC4GD1', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.018461", "end": "2019-10-01 02:18:29.258937", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "labels": [], "masters": [], "uuids": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:28.240476", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:29.386 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:29 +0000 (0:00:08.416) 0:07:30.723 ******* 2019-10-01T02:18:29.884 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001985300'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N2108YSE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdc'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdc || sgdisk --zap-all /dev/sdc", "delta": "0:00:01.021832", "end": "2019-10-01 02:18:29.825316", "item": {"key": "sdc", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001985300"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2108YSE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:28.803484", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:30.167 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-01T02:18:30.169 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010070802'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdd'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdd bs=1M count=110", "delta": "0:00:00.251955", "end": "2019-10-01 02:18:30.121434", "item": {"key": "sdd", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010070802"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:29.869479", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.24372 s, 473 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.24372 s, 473 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:30.853 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2070345100'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N20GDSAE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sde'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sde bs=1M count=110", "delta": "0:00:00.254730", "end": "2019-10-01 02:18:30.814590", "item": {"key": "sde", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2070345100"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N20GDSAE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:30.559860", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.246969 s, 467 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.246969 s, 467 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:31.326 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001535300'], u'uuids': [u'b46ae0b3-5463-4677-893e-f532ede61502']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N2103U3E', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdh'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/sdh || sgdisk --zap-all /dev/sdh", "delta": "0:00:01.020925", "end": "2019-10-01 02:18:31.258630", "item": {"key": "sdh", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001535300"], "labels": [], "masters": [], "uuids": ["b46ae0b3-5463-4677-893e-f532ede61502"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2103U3E", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:30.237705", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-01T02:18:31.387 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:31 +0000 (0:00:02.000) 0:07:32.724 ******* 2019-10-01T02:18:31.407 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-01T02:18:31.409 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Seagate', u'links': {u'masters': [u'dm-0'], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'6VPC4GD1', u'holders': [u'mpatha'], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST31000528AS', u'partitions': {}}, 'key': u'sdf'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdf bs=1M count=110", "delta": "0:00:00.077218", "end": "2019-10-01 02:18:31.348970", "item": {"key": "sdf", "value": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": [], "labels": [], "masters": ["dm-0"], "uuids": []}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:31.271752", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0692252 s, 1.7 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0692252 s, 1.7 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:32.108 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010866207'], u'uuids': [u'a560672c-38f4-4d12-9eb7-f21f6fd80dba']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdg'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdg bs=1M count=110", "delta": "0:00:00.276691", "end": "2019-10-01 02:18:32.056877", "item": {"key": "sdg", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010866207"], "labels": [], "masters": [], "uuids": ["a560672c-38f4-4d12-9eb7-f21f6fd80dba"]}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:31.780186", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.269659 s, 428 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.269659 s, 428 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:32.121 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001755c00'], u'uuids': [u'08894f69-ea05-45a2-a056-85d846cce4c6']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N210EWLE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdd'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdd bs=1M count=110", "delta": "0:00:00.280270", "end": "2019-10-01 02:18:32.047354", "item": {"key": "sdd", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001755c00"], "labels": [], "masters": [], "uuids": ["08894f69-ea05-45a2-a056-85d846cce4c6"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210EWLE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:31.767084", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.27281 s, 423 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.27281 s, 423 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:32.123 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010960009'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010960009-part1'], u'uuids': [u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010960009"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d2010960009-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:32.816 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20a0443200'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N20ZTT2C', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sde'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sde bs=1M count=110", "delta": "0:00:00.247319", "end": "2019-10-01 02:18:32.773765", "item": {"key": "sde", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20a0443200"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N20ZTT2C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:32.526446", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.239795 s, 481 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.239795 s, 481 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:33.498 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001373100'], u'uuids': [u'ac2b8ca6-941f-4b45-90a5-6efdddf15630']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N21073AC', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdf'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdf bs=1M count=110", "delta": "0:00:00.256387", "end": "2019-10-01 02:18:33.454865", "item": {"key": "sdf", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001373100"], "labels": [], "masters": [], "uuids": ["ac2b8ca6-941f-4b45-90a5-6efdddf15630"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N21073AC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:33.198478", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.248753 s, 464 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.248753 s, 464 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:34.133 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010572303'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdb'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdb bs=1M count=110", "delta": "0:00:01.568352", "end": "2019-10-01 02:18:34.086290", "item": {"key": "sdb", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010572303"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:32.517938", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 1.55949 s, 74.0 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 1.55949 s, 74.0 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:34.189 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001475400'], u'uuids': [u'adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N210WTDE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdg'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdg bs=1M count=110", "delta": "0:00:00.255590", "end": "2019-10-01 02:18:34.140807", "item": {"key": "sdg", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001475400"], "labels": [], "masters": [], "uuids": ["adcdd0f8-4ee2-4d2d-8b33-5dcfd29d91b9"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N210WTDE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:33.885217", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.247169 s, 467 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.247169 s, 467 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:34.207 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20114a3400'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N211JT4C', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20114a3400-part1'], u'uuids': [u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": { 2019-10-01T02:18:34.210 INFO:teuthology.task.ansible.out:"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20114a3400"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {"sda1": {"holders": [], "links": {"ids": ["scsi-2001b4d20114a3400-part1"], "labels": [], "masters": [], "uuids": ["254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211JT4C", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:34.871 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'WDC', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2010865804'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'WD', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'WD1002F9YZ-09H1J', u'partitions': {}}, 'key': u'sdc'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdc bs=1M count=110", "delta": "0:00:00.265141", "end": "2019-10-01 02:18:34.813043", "item": {"key": "sdc", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2010865804"], "labels": [], "masters": [], "uuids": []}, "model": "WD1002F9YZ-09H1J", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "WD", "size": "931.51 GB", "support_discard": "0", "vendor": "WDC", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:34.547902", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.257101 s, 449 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.257101 s, 449 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:34.902 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d20115e3600'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9J0N211NUVC', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdb'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdb bs=1M count=110", "delta": "0:00:00.289152", "end": "2019-10-01 02:18:34.851243", "item": {"key": "sdb", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d20115e3600"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9J0N211NUVC", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:34.562091", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.280588 s, 411 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.280588 s, 411 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:35.658 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001985300'], u'uuids': []}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N2108YSE', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdc'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdc bs=1M count=110", "delta": "0:00:00.255931", "end": "2019-10-01 02:18:35.608210", "item": {"key": "sdc", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001985300"], "labels": [], "masters": [], "uuids": []}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2108YSE", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:35.352279", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.248292 s, 465 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.248292 s, 465 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:36.331 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Hitachi', u'links': {u'masters': [], u'labels': [], u'ids': [u'scsi-2001b4d2001535300'], u'uuids': [u'b46ae0b3-5463-4677-893e-f532ede61502']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'JPW9K0N2103U3E', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'HUA722010CLA330', u'partitions': {}}, 'key': u'sdh'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdh bs=1M count=110", "delta": "0:00:00.241340", "end": "2019-10-01 02:18:36.287046", "item": {"key": "sdh", "value": {"holders": [], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2001535300"], "labels": [], "masters": [], "uuids": ["b46ae0b3-5463-4677-893e-f532ede61502"]}, "model": "HUA722010CLA330", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "JPW9K0N2103U3E", "size": "931.51 GB", "support_discard": "0", "vendor": "Hitachi", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:36.045706", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.233104 s, 495 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.233104 s, 495 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:36.480 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:36 +0000 (0:00:05.093) 0:07:37.817 ******* 2019-10-01T02:18:36.501 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'Seagate', u'links': {u'masters': [u'dm-0'], u'labels': [], u'ids': [u'scsi-2001b4d2000000000'], u'uuids': [u'11dd6939-301d-45ff-b781-d2460ec22316']}, u'sas_device_handle': None, u'host': u'RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller', u'support_discard': u'0', u'serial': u'9VP9NVCL', u'holders': [u'mpatha'], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST31000528AS', u'partitions': {}}, 'key': u'sdh'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/sdh bs=1M count=110", "delta": "0:00:01.070667", "end": "2019-10-01 02:18:36.328691", "item": {"key": "sdh", "value": {"holders": ["mpatha"], "host": "RAID bus controller: Areca Technology Corp. ARC-1680 series PCIe to SAS/SATA 3Gb RAID Controller", "links": {"ids": ["scsi-2001b4d2000000000"], "labels": [], "masters": ["dm-0"], "uuids": ["11dd6939-301d-45ff-b781-d2460ec22316"]}, "model": "ST31000528AS", "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "9VP9NVCL", "size": "931.51 GB", "support_discard": "0", "vendor": "Seagate", "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:35.258024", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0612314 s, 1.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0612314 s, 1.9 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:36.994 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-01T02:18:36.996 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.017764", "end": "2019-10-01 02:18:36.946597", "rc": 0, "start": "2019-10-01 02:18:36.928833", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:37.029 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:37 +0000 (0:00:00.548) 0:07:38.366 ******* 2019-10-01T02:18:37.638 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [mira064.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.108966", "end": "2019-10-01 02:18:37.580739", "rc": 0, "start": "2019-10-01 02:18:37.471773", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:37.704 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:37 +0000 (0:00:00.671) 0:07:39.037 ******* 2019-10-01T02:18:37.743 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-01T02:18:37.810 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:37 +0000 (0:00:00.109) 0:07:39.147 ******* 2019-10-01T02:18:37.860 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:37.904 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:37 +0000 (0:00:00.089) 0:07:39.236 ******* 2019-10-01T02:18:37.925 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-01T02:18:37.930 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:37.971 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:37 +0000 (0:00:00.072) 0:07:39.309 ******* 2019-10-01T02:18:38.008 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:38.073 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:38 +0000 (0:00:00.097) 0:07:39.406 ******* 2019-10-01T02:18:38.111 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-01T02:18:38.113 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:38.170 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:38 +0000 (0:00:00.101) 0:07:39.508 ******* 2019-10-01T02:18:38.673 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-01T02:18:38.674 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:18:38.722 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:38 +0000 (0:00:00.551) 0:07:40.059 ******* 2019-10-01T02:18:38.785 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-01T02:18:38.788 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={u'block_used': 12996189, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328295, u'size_available': 931136696320, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328295, "block_size": 4096, "block_total": 240324484, "block_used": 12996189, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136696320, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:38.840 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:38 +0000 (0:00:00.117) 0:07:40.177 ******* 2019-10-01T02:18:38.881 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:38.883 INFO:teuthology.task.ansible.out:TASK [testnode : Combine list of non-root disks] ******************************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:38.935 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:38 +0000 (0:00:00.095) 0:07:40.273 ******* 2019-10-01T02:18:38.978 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:18:38.980 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.058 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.123) 0:07:40.396 ******* 2019-10-01T02:18:39.098 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.152 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.093) 0:07:40.490 ******* 2019-10-01T02:18:39.197 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-01T02:18:39.199 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.255 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.102) 0:07:40.593 ******* 2019-10-01T02:18:39.295 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.335 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.079) 0:07:40.672 ******* 2019-10-01T02:18:39.393 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.439 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.103) 0:07:40.776 ******* 2019-10-01T02:18:39.478 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-01T02:18:39.479 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.532 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.089) 0:07:40.866 ******* 2019-10-01T02:18:39.581 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-01T02:18:39.582 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.627 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.096) 0:07:40.963 ******* 2019-10-01T02:18:39.675 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.713 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.087) 0:07:41.050 ******* 2019-10-01T02:18:39.766 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-01T02:18:39.773 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.820 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.106) 0:07:41.157 ******* 2019-10-01T02:18:39.864 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:39.922 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:39 +0000 (0:00:00.101) 0:07:41.259 ******* 2019-10-01T02:18:39.962 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-10-01T02:18:39.964 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.019 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.094) 0:07:41.353 ******* 2019-10-01T02:18:40.092 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on {{ var_lib_partition }}] ************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.168 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.150) 0:07:41.504 ******* 2019-10-01T02:18:40.240 INFO:teuthology.task.ansible.out: TASK [testnode : Mount {{ var_lib_partition }} to /var/lib/ceph] *************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.283 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.114) 0:07:41.619 ******* 2019-10-01T02:18:40.323 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.356 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.073) 0:07:41.693 ******* 2019-10-01T02:18:40.399 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-01T02:18:40.400 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.452 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.093) 0:07:41.787 ******* 2019-10-01T02:18:40.482 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-01T02:18:40.484 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.528 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.075) 0:07:41.862 ******* 2019-10-01T02:18:40.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-01T02:18:40.569 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.600 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.075) 0:07:41.937 ******* 2019-10-01T02:18:40.643 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-01T02:18:40.646 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.689 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.085) 0:07:42.023 ******* 2019-10-01T02:18:40.735 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.761 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.075) 0:07:42.098 ******* 2019-10-01T02:18:40.790 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-01T02:18:40.792 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:40.833 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:40 +0000 (0:00:00.072) 0:07:42.171 ******* 2019-10-01T02:18:41.658 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-01T02:18:41.659 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-10-01T02:18:41.722 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:41 +0000 (0:00:00.886) 0:07:43.057 ******* 2019-10-01T02:18:41.760 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:41.804 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:41 +0000 (0:00:00.083) 0:07:43.141 ******* 2019-10-01T02:18:41.832 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:41.893 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:41 +0000 (0:00:00.083) 0:07:43.225 ******* 2019-10-01T02:18:43.198 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-01T02:18:43.199 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-10-01T02:18:43.259 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:43 +0000 (0:00:01.370) 0:07:44.596 ******* 2019-10-01T02:18:43.310 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:43.379 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:43 +0000 (0:00:00.117) 0:07:44.713 ******* 2019-10-01T02:18:44.079 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-01T02:18:44.082 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount syslog.target -.mount system.slice sntp.service systemd-journald.socket ntpdate.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target chronyd.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615" 2019-10-01T02:18:44.084 INFO:teuthology.task.ansible.out:, "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": " 2019-10-01T02:18:44.085 INFO:teuthology.task.ansible.out:50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-01T02:18:44.135 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:44 +0000 (0:00:00.757) 0:07:45.471 ******* 2019-10-01T02:18:45.423 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-01T02:18:45.424 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-8.el7.noarch providing python-pip is already installed"]} 2019-10-01T02:18:45.467 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:45 +0000 (0:00:01.329) 0:07:46.801 ******* 2019-10-01T02:18:45.492 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-01T02:18:45.494 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:45.536 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:45 +0000 (0:00:00.068) 0:07:46.870 ******* 2019-10-01T02:18:46.002 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-01T02:18:46.003 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "gid": 1206, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-10-01T02:18:46.067 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:46 +0000 (0:00:00.534) 0:07:47.404 ******* 2019-10-01T02:18:46.240 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-01T02:18:46.249 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-mpatha', u'dm-uuid-mpath-2001b4d2000000000'], u'uuids': [u'7e986d2e-887e-4bc5-b72b-2c2e4214f494']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'serial': u'6VPC4GD1', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:09.365156", "end": "2019-10-01 02:18:46.193108", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-mpatha", "dm-uuid-mpath-2001b4d2000000000"], "labels": [], "masters": [], "uuids": ["7e986d2e-887e-4bc5-b72b-2c2e4214f494"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "6VPC4GD1", "size": "931.51 GB", "support_discard": "0", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-01 02:18:36.827952", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 9.35753 s, 12.3 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 9.35753 s, 12.3 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:46.344 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:46 +0000 (0:00:00.276) 0:07:47.681 ******* 2019-10-01T02:18:47.374 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [mira064.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1206, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-10-01T02:18:47.410 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:47 +0000 (0:00:01.066) 0:07:48.748 ******* 2019-10-01T02:18:47.433 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:47.649 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-01T02:18:47.650 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.851847", "end": "2019-10-01 02:18:47.600802", "rc": 0, "start": "2019-10-01 02:18:46.748955", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:47.739 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:47 +0000 (0:00:00.328) 0:07:49.077 ******* 2019-10-01T02:18:48.309 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-01T02:18:48.311 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.114429", "end": "2019-10-01 02:18:48.267925", "rc": 0, "start": "2019-10-01 02:18:48.153496", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:48.356 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.616) 0:07:49.693 ******* 2019-10-01T02:18:48.385 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-01T02:18:48.424 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.068) 0:07:49.761 ******* 2019-10-01T02:18:48.463 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-01T02:18:48.466 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:48.521 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.096) 0:07:49.858 ******* 2019-10-01T02:18:48.559 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-01T02:18:48.560 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:48.609 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.083) 0:07:49.941 ******* 2019-10-01T02:18:48.644 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:48.645 INFO:teuthology.task.ansible.out:TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:48.705 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.098) 0:07:50.040 ******* 2019-10-01T02:18:48.735 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-01T02:18:48.737 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:48.774 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:48 +0000 (0:00:00.071) 0:07:50.112 ******* 2019-10-01T02:18:49.248 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-01T02:18:49.249 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-01T02:18:49.304 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.530) 0:07:50.642 ******* 2019-10-01T02:18:49.356 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-01T02:18:49.357 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={u'block_used': 12996204, u'uuid': u'254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227328280, u'size_available': 931136634880, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60967551, u'device': u'/dev/sda1', u'inode_used': 87425, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227328280, "block_size": 4096, "block_total": 240324484, "block_used": 12996204, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60967551, "inode_total": 61054976, "inode_used": 87425, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 931136634880, "size_total": 984369086464, "uuid": "254b5f97-a3c2-4fa0-9cdb-62cf4484d4a7"}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.404 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.097) 0:07:50.739 ******* 2019-10-01T02:18:49.444 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.493 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.088) 0:07:50.828 ******* 2019-10-01T02:18:49.523 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:18:49.526 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.568 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.071) 0:07:50.900 ******* 2019-10-01T02:18:49.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:18:49.613 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.666 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.103) 0:07:51.004 ******* 2019-10-01T02:18:49.699 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-01T02:18:49.701 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.733 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.066) 0:07:51.070 ******* 2019-10-01T02:18:49.770 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-01T02:18:49.772 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.805 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.071) 0:07:51.142 ******* 2019-10-01T02:18:49.845 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-01T02:18:49.846 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.847 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [mira064.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.2.3 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.2.3 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-01T02:18:49.882 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.077) 0:07:51.220 ******* 2019-10-01T02:18:49.919 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.035) 0:07:51.256 ******* 2019-10-01T02:18:49.925 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-01T02:18:49.926 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:49.968 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:49 +0000 (0:00:00.049) 0:07:51.305 ******* 2019-10-01T02:18:49.974 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-01T02:18:49.976 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.014 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.045) 0:07:51.351 ******* 2019-10-01T02:18:50.021 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-01T02:18:50.022 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.061 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.045) 0:07:51.398 ******* 2019-10-01T02:18:50.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-01T02:18:50.070 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp4s0"}, "changed": false} 2019-10-01T02:18:50.111 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.047) 0:07:51.445 ******* 2019-10-01T02:18:50.120 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-01T02:18:50.120 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.157 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.048) 0:07:51.494 ******* 2019-10-01T02:18:50.184 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.224 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.063) 0:07:51.558 ******* 2019-10-01T02:18:50.247 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-01T02:18:50.249 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.281 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.060) 0:07:51.619 ******* 2019-10-01T02:18:50.322 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.370 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.087) 0:07:51.706 ******* 2019-10-01T02:18:50.398 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on {{ var_lib_partition }}] ************* 2019-10-01T02:18:50.400 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.451 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.081) 0:07:51.788 ******* 2019-10-01T02:18:50.483 INFO:teuthology.task.ansible.out: TASK [testnode : Mount {{ var_lib_partition }} to /var/lib/ceph] *************** 2019-10-01T02:18:50.485 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.518 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.067) 0:07:51.856 ******* 2019-10-01T02:18:50.545 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-01T02:18:50.547 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.583 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.064) 0:07:51.920 ******* 2019-10-01T02:18:50.589 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-01T02:18:50.590 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-10-01T02:18:50.630 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.047) 0:07:51.967 ******* 2019-10-01T02:18:50.637 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-01T02:18:50.638 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.679 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.048) 0:07:52.016 ******* 2019-10-01T02:18:50.688 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-01T02:18:50.690 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.726 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.046) 0:07:52.063 ******* 2019-10-01T02:18:50.735 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.745 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:50.777 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.050) 0:07:52.114 ******* 2019-10-01T02:18:50.783 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-01T02:18:50.785 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-01T02:18:50.819 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.042) 0:07:52.157 ******* 2019-10-01T02:18:50.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-01T02:18:50.827 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.863 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.043) 0:07:52.200 ******* 2019-10-01T02:18:50.871 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.904 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.041) 0:07:52.241 ******* 2019-10-01T02:18:50.911 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-01T02:18:50.913 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:50.965 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:50 +0000 (0:00:00.060) 0:07:52.302 ******* 2019-10-01T02:18:50.995 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-01T02:18:50.996 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:51.030 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:51 +0000 (0:00:00.065) 0:07:52.367 ******* 2019-10-01T02:18:51.069 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-01T02:18:51.071 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:51.118 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:51 +0000 (0:00:00.087) 0:07:52.455 ******* 2019-10-01T02:18:51.377 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-10-01T02:18:51.379 INFO:teuthology.task.ansible.out:changed: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-01T02:18:51.416 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:51 +0000 (0:00:00.298) 0:07:52.753 ******* 2019-10-01T02:18:51.873 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-01T02:18:51.874 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-10-01T02:18:51.875 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [mira064.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-01T02:18:51.923 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:51 +0000 (0:00:00.507) 0:07:53.261 ******* 2019-10-01T02:18:51.960 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:51 +0000 (0:00:00.036) 0:07:53.297 ******* 2019-10-01T02:18:51.965 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-10-01T02:18:51.967 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.017 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.057) 0:07:53.354 ******* 2019-10-01T02:18:52.027 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-01T02:18:52.028 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.066 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.048) 0:07:53.403 ******* 2019-10-01T02:18:52.071 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-10-01T02:18:52.073 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.127 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.061) 0:07:53.465 ******* 2019-10-01T02:18:52.134 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-01T02:18:52.136 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item=enp4s0) => {"ansible_loop_var": "item", "changed": false, "item": "enp4s0", "skip_reason": "Conditional result was False"} skipping: [mira064.front.sepia.ceph.com] => (item=enp5s0) => {"ansible_loop_var": "item", "changed": false, "item": "enp5s0", "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.187 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.059) 0:07:53.524 ******* 2019-10-01T02:18:52.230 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-01T02:18:52.231 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.243 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp4s0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp4s0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.244 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp5s0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp5s0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.299 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.112) 0:07:53.636 ******* 2019-10-01T02:18:52.326 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:18:52.328 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.375 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.076) 0:07:53.712 ******* 2019-10-01T02:18:52.414 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-01T02:18:52.416 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.471 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.096) 0:07:53.809 ******* 2019-10-01T02:18:52.516 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-01T02:18:52.517 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.573 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.101) 0:07:53.910 ******* 2019-10-01T02:18:52.623 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-01T02:18:52.625 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.678 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.104) 0:07:54.015 ******* 2019-10-01T02:18:52.717 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-01T02:18:52.719 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.772 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.094) 0:07:54.110 ******* 2019-10-01T02:18:52.811 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-01T02:18:52.813 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.854 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.080) 0:07:54.190 ******* 2019-10-01T02:18:52.892 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-10-01T02:18:52.893 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:52.976 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:52 +0000 (0:00:00.123) 0:07:54.313 ******* 2019-10-01T02:18:53.004 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-01T02:18:53.006 INFO:teuthology.task.ansible.out:skipping: [mira064.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:53.043 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:53 +0000 (0:00:00.066) 0:07:54.380 ******* 2019-10-01T02:18:53.480 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-10-01T02:18:53.523 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:53 +0000 (0:00:00.480) 0:07:54.861 ******* 2019-10-01T02:18:53.556 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [mira064.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.015840", "end": "2019-10-01 02:18:53.486661", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-01 02:18:53.470821", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Create the chrony.conf file] ********************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:53.606 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:53 +0000 (0:00:00.082) 0:07:54.943 ******* 2019-10-01T02:18:54.011 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-01T02:18:54.011 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.009340", "end": "2019-10-01 02:18:53.962072", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-01 02:18:53.952732", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:54.454 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-01T02:18:54.456 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target system.slice -.mount sntp.service systemd-journald.socket syslog.target tmp.mount ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target chronyd.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615" 2019-10-01T02:18:54.458 INFO:teuthology.task.ansible.out:, "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "63420", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "63420", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": " 2019-10-01T02:18:54.459 INFO:teuthology.task.ansible.out:50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-10-01T02:18:54.520 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:54 +0000 (0:00:00.913) 0:07:55.857 ******* 2019-10-01T02:18:54.521 INFO:teuthology.task.ansible.out: 2019-10-01T02:18:55.446 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-01T02:18:55.447 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:01.010234", "end": "2019-10-01 02:18:55.404212", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-01 02:18:54.393978", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:55.803 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-01T02:18:55.804 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-8.el7.noarch providing python-pip is already installed"]} 2019-10-01T02:18:55.844 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:55 +0000 (0:00:01.324) 0:07:57.182 ******* 2019-10-01T02:18:55.871 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-01T02:18:55.873 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:55.930 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:55 +0000 (0:00:00.085) 0:07:57.267 ******* 2019-10-01T02:18:56.395 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-01T02:18:56.396 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "gid": 1206, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-10-01T02:18:56.431 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:56 +0000 (0:00:00.501) 0:07:57.768 ******* 2019-10-01T02:18:56.887 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-01T02:18:56.888 INFO:teuthology.task.ansible.out:ok: [mira064.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.011316", "end": "2019-10-01 02:18:56.829396", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-01 02:18:55.818080", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:18:56.931 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:56 +0000 (0:00:00.500) 0:07:58.269 ******* 2019-10-01T02:18:56.958 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-01T02:18:56.995 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:56 +0000 (0:00:00.063) 0:07:58.332 ******* 2019-10-01T02:18:57.451 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [mira064.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-01T02:18:57.703 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [mira063.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1206, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-10-01T02:18:57.736 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:57 +0000 (0:00:00.741) 0:07:59.074 ******* 2019-10-01T02:18:57.757 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:18:59.477 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-01T02:18:59.481 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.2.3 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.2.3 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-01T02:18:59.521 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:59 +0000 (0:00:01.784) 0:08:00.859 ******* 2019-10-01T02:18:59.548 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-01T02:18:59.550 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:18:59.583 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:59 +0000 (0:00:00.061) 0:08:00.920 ******* 2019-10-01T02:18:59.620 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-01T02:18:59.622 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp4s0"}, "changed": false} 2019-10-01T02:18:59.670 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:18:59 +0000 (0:00:00.086) 0:08:01.007 ******* 2019-10-01T02:19:00.127 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-01T02:19:00.129 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-10-01T02:19:00.169 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:00 +0000 (0:00:00.495) 0:08:01.503 ******* 2019-10-01T02:19:00.202 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-01T02:19:00.204 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:00.243 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:00 +0000 (0:00:00.073) 0:08:01.576 ******* 2019-10-01T02:19:00.259 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-01T02:19:00.276 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-01T02:19:00.278 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-01T02:19:00.326 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:00 +0000 (0:00:00.087) 0:08:01.663 ******* 2019-10-01T02:19:00.367 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:00.411 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:00 +0000 (0:00:00.084) 0:08:01.748 ******* 2019-10-01T02:19:00.867 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-01T02:19:00.905 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:00 +0000 (0:00:00.493) 0:08:02.242 ******* 2019-10-01T02:19:01.387 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [mira063.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-01T02:19:01.435 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.529) 0:08:02.771 ******* 2019-10-01T02:19:01.471 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-01T02:19:01.473 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.513 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.079) 0:08:02.850 ******* 2019-10-01T02:19:01.552 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-01T02:19:01.554 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [mira063.front.sepia.ceph.com] => (item=enp4s0) => {"ansible_loop_var": "item", "changed": false, "item": "enp4s0", "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.565 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item=enp5s0) => {"ansible_loop_var": "item", "changed": false, "item": "enp5s0", "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.604 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.090) 0:08:02.941 ******* 2019-10-01T02:19:01.640 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-01T02:19:01.642 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.653 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp4s0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp4s0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.654 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp5s0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp5s0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.702 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.097) 0:08:03.039 ******* 2019-10-01T02:19:01.728 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-01T02:19:01.730 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.774 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.072) 0:08:03.111 ******* 2019-10-01T02:19:01.804 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.856 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.082) 0:08:03.193 ******* 2019-10-01T02:19:01.896 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-01T02:19:01.898 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:01.946 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:01 +0000 (0:00:00.089) 0:08:03.283 ******* 2019-10-01T02:19:01.974 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-01T02:19:01.976 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:02.012 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:02 +0000 (0:00:00.066) 0:08:03.350 ******* 2019-10-01T02:19:02.053 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-01T02:19:02.055 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:02.091 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:02 +0000 (0:00:00.079) 0:08:03.429 ******* 2019-10-01T02:19:02.119 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-01T02:19:02.121 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:02.163 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:02 +0000 (0:00:00.069) 0:08:03.498 ******* 2019-10-01T02:19:02.200 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-10-01T02:19:02.202 INFO:teuthology.task.ansible.out:skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:02.239 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:02 +0000 (0:00:00.077) 0:08:03.576 ******* 2019-10-01T02:19:02.282 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [mira063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-01T02:19:02.317 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:02 +0000 (0:00:00.079) 0:08:03.655 ******* 2019-10-01T02:19:02.823 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-01T02:19:02.828 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.019021", "end": "2019-10-01 02:19:02.754366", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-01 02:19:02.735345", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:19:03.285 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.009269", "end": "2019-10-01 02:19:03.231207", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-01 02:19:03.221938", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:19:03.747 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.008530", "end": "2019-10-01 02:19:03.708174", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-01 02:19:03.699644", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-01T02:19:04.178 INFO:teuthology.task.ansible.out:ok: [mira063.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.010090", "end": "2019-10-01 02:19:04.130801", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-01 02:19:04.120711", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-01T02:19:04.215 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:04 +0000 (0:00:01.897) 0:08:05.553 ******* 2019-10-01T02:19:04.241 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-01T02:19:04.278 INFO:teuthology.task.ansible.out:Tuesday 01 October 2019 02:19:04 +0000 (0:00:00.062) 0:08:05.615 ******* 2019-10-01T02:19:04.696 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-01T02:19:04.697 INFO:teuthology.task.ansible.out:changed: [mira063.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-10-01T02:19:04.814 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2019-10-01T02:19:04.815 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-10-01T02:19:04.818 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2019-10-01T02:19:04.819 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-10-01T02:19:04.821 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* mira063.front.sepia.ceph.com : ok=105 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 mira064.front.sepia.ceph.com : ok=105 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Tuesday 01 October 2019 02:19:04 +0000 (0:00:00.542) 0:08:06.158 ******* 2019-10-01T02:19:04.822 INFO:teuthology.task.ansible.out:=============================================================================== 2019-10-01T02:19:04.824 INFO:teuthology.task.ansible.out:testnode : Install packages -------------------------------------------- 63.53s users : Create all admin users with sudo access. ----------------------- 54.84s users : Update authorized_keys using the keys repo --------------------- 51.38s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.48s common : Install nrpe package and dependencies (RHEL/CentOS) ----------- 29.71s testnode : Ensure ceph packages are not present. ----------------------- 23.15s common : nrpe - Remove SELinux policy package -------------------------- 13.87s common : Make sure smartmontools is installed -------------------------- 13.85s common : nrpe - Load SELinux policy package ---------------------------- 13.80s users : Remove revoked users ------------------------------------------- 12.39s testnode : Ensure ceph dependency packages are not present. ------------- 8.91s testnode : Zap all non-root disks --------------------------------------- 8.42s testnode : Blow away lingering OSD data and FSIDs ----------------------- 5.09s testnode : Removing saved yum transactions ------------------------------ 5.05s ansible-managed : Add authorized keys for the ansible user. ------------- 3.88s users : Update authorized_keys for each user with literal keys ---------- 3.27s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 3.05s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 3.00s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 2.57s Check for /usr/bin/python ----------------------------------------------- 2.55s 2019-10-01T02:19:05.112 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@mira063.front.sepia.ceph.com'), Remote(name='ubuntu@mira064.front.sepia.ceph.com')] 2019-10-01T02:19:05.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:19:05.115 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira063.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:19:05.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:05.222 INFO:teuthology.orchestra.run.mira063:> true 2019-10-01T02:19:05.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-01T02:19:05.657 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira064.front.sepia.ceph.com', 'timeout': 60} 2019-10-01T02:19:05.792 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:05.793 INFO:teuthology.orchestra.run.mira064:> true 2019-10-01T02:19:06.238 INFO:teuthology.run_tasks:Running task clock... 2019-10-01T02:19:06.254 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-10-01T02:19:06.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:06.255 INFO:teuthology.orchestra.run.mira063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-01T02:19:06.437 INFO:teuthology.orchestra.run.mira063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-10-01T02:19:16.580 INFO:teuthology.orchestra.run.mira063.stdout:ntpd: time slew +0.001330s 2019-10-01T02:19:16.635 INFO:teuthology.orchestra.run.mira063.stderr:Failed to start ntp.service: Unit not found. 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout: remote refid st t when poll reach delay offset jitter 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout:============================================================================== 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:16.748 INFO:teuthology.orchestra.run.mira063.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:16.749 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:16.750 INFO:teuthology.orchestra.run.mira064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-01T02:19:16.921 INFO:teuthology.orchestra.run.mira064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-10-01T02:19:27.055 INFO:teuthology.orchestra.run.mira064.stdout:ntpd: time slew +0.001093s 2019-10-01T02:19:27.104 INFO:teuthology.orchestra.run.mira064.stderr:Failed to start ntp.service: Unit not found. 2019-10-01T02:19:27.215 INFO:teuthology.orchestra.run.mira064.stdout: remote refid st t when poll reach delay offset jitter 2019-10-01T02:19:27.216 INFO:teuthology.orchestra.run.mira064.stdout:============================================================================== 2019-10-01T02:19:27.216 INFO:teuthology.orchestra.run.mira064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:27.216 INFO:teuthology.orchestra.run.mira064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:27.216 INFO:teuthology.orchestra.run.mira064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:27.217 INFO:teuthology.orchestra.run.mira064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-01T02:19:27.218 INFO:teuthology.run_tasks:Running task install... 2019-10-01T02:19:27.236 DEBUG:teuthology.task.install:project ceph 2019-10-01T02:19:27.236 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027'}} 2019-10-01T02:19:27.236 DEBUG:teuthology.task.install:config {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027'} 2019-10-01T02:19:27.236 INFO:teuthology.task.install:Using flavor: basic 2019-10-01T02:19:27.244 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-10-01T02:19:27.245 INFO:teuthology.task.install:extra packages: [] 2019-10-01T02:19:27.245 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:27.245 INFO:teuthology.orchestra.run.mira063:> sudo lsb_release -is 2019-10-01T02:19:27.467 INFO:teuthology.orchestra.run.mira063.stdout:CentOS 2019-10-01T02:19:27.470 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:19:27.470 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:27.470 INFO:teuthology.orchestra.run.mira064:> sudo lsb_release -is 2019-10-01T02:19:27.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2019-10-01T02:19:27.471 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:27.472 INFO:teuthology.orchestra.run.mira063:> sudo lsb_release -is 2019-10-01T02:19:27.680 INFO:teuthology.orchestra.run.mira063.stdout:CentOS 2019-10-01T02:19:27.683 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:19:27.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=2c0b2169f6303e11ab6bed736272e5e2d23ab027 2019-10-01T02:19:27.696 INFO:teuthology.orchestra.run.mira064.stdout:CentOS 2019-10-01T02:19:27.698 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:19:27.698 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2019-10-01T02:19:27.699 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:27.699 INFO:teuthology.orchestra.run.mira064:> sudo lsb_release -is 2019-10-01T02:19:27.845 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2019-09-30-1336/2c0b2169f6303e11ab6bed736272e5e2d23ab027/centos/7/flavors/default/ 2019-10-01T02:19:27.846 INFO:teuthology.task.install.rpm:Package version is 15.0.0-5673.g2c0b216 2019-10-01T02:19:27.907 INFO:teuthology.orchestra.run.mira064.stdout:CentOS 2019-10-01T02:19:27.911 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:19:27.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=2c0b2169f6303e11ab6bed736272e5e2d23ab027 2019-10-01T02:19:27.967 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:27.968 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-10-01T02:19:28.068 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2019-09-30-1336/2c0b2169f6303e11ab6bed736272e5e2d23ab027/centos/7/flavors/default/ 2019-10-01T02:19:28.068 INFO:teuthology.task.install.rpm:Package version is 15.0.0-5673.g2c0b216 2019-10-01T02:19:28.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-10-01T02:19:28.134 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:28.134 INFO:teuthology.orchestra.run.mira063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2c0b2169f6303e11ab6bed736272e5e2d23ab027/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-10-01T02:19:28.193 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:28.193 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-10-01T02:19:28.294 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:28.295 INFO:teuthology.orchestra.run.mira063:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-10-01T02:19:28.357 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-10-01T02:19:28.358 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:28.358 INFO:teuthology.orchestra.run.mira064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2c0b2169f6303e11ab6bed736272e5e2d23ab027/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-10-01T02:19:28.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:28.450 INFO:teuthology.orchestra.run.mira063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-10-01T02:19:28.522 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:28.522 INFO:teuthology.orchestra.run.mira064:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-10-01T02:19:28.603 INFO:teuthology.orchestra.run.mira063.stdout:check_obsoletes = 1 2019-10-01T02:19:28.605 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:28.605 INFO:teuthology.orchestra.run.mira063:> sudo yum clean all 2019-10-01T02:19:28.683 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:28.684 INFO:teuthology.orchestra.run.mira064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-10-01T02:19:28.841 INFO:teuthology.orchestra.run.mira064.stdout:check_obsoletes = 1 2019-10-01T02:19:28.844 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:28.844 INFO:teuthology.orchestra.run.mira064:> sudo yum clean all 2019-10-01T02:19:28.974 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:19:29.049 INFO:teuthology.orchestra.run.mira063.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-10-01T02:19:29.049 INFO:teuthology.orchestra.run.mira063.stdout: : lab-extras updates 2019-10-01T02:19:29.077 INFO:teuthology.orchestra.run.mira063.stdout:Cleaning up list of fastest mirrors 2019-10-01T02:19:29.095 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:19:29.096 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-radosgw 2019-10-01T02:19:29.229 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:19:29.305 INFO:teuthology.orchestra.run.mira064.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-10-01T02:19:29.305 INFO:teuthology.orchestra.run.mira064.stdout: : lab-extras updates 2019-10-01T02:19:29.332 INFO:teuthology.orchestra.run.mira064.stdout:Cleaning up list of fastest mirrors 2019-10-01T02:19:29.351 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:19:29.352 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-radosgw 2019-10-01T02:19:29.475 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:19:29.550 INFO:teuthology.orchestra.run.mira063.stdout:Determining fastest mirrors 2019-10-01T02:19:29.692 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:19:29.767 INFO:teuthology.orchestra.run.mira064.stdout:Determining fastest mirrors 2019-10-01T02:19:30.122 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:19:30.123 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:19:30.124 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:19:30.125 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:19:30.398 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:19:30.400 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:19:30.400 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:19:30.401 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:19:47.505 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:19:47.535 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:19:48.123 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:19:48.123 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:19:48.123 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.149 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.150 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:19:48.151 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:19:48.151 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.176 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.360 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.364 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.366 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librados2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.369 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.369 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.373 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.375 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libceph-common.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.376 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:19:48.376 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-base.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.387 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.404 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.405 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.405 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.405 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.405 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librados2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.406 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.406 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.406 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.406 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libceph-common.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.407 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:19:48.407 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-base.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.423 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.423 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.423 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.423 INFO:teuthology.orchestra.run.mira063.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-10-01T02:19:48.424 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-10-01T02:19:48.424 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-10-01T02:19:48.424 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librados2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be an update 2019-10-01T02:19:48.424 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librgw2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.425 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:19:48.425 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.426 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.426 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.426 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.428 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.429 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.429 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.431 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.431 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.455 INFO:teuthology.orchestra.run.mira064.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-10-01T02:19:48.455 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-10-01T02:19:48.455 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-10-01T02:19:48.456 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librados2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be an update 2019-10-01T02:19:48.457 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librgw2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.457 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:19:48.457 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.457 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.457 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.458 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.458 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-rados = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.458 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.458 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.458 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.459 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.459 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.459 INFO:teuthology.orchestra.run.mira063.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-10-01T02:19:48.459 INFO:teuthology.orchestra.run.mira063.stdout:---> Package libcephfs2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.460 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-10-01T02:19:48.460 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librbd1.x86_64 2:15.0.0-5673.g2c0b216.el7 will be an update 2019-10-01T02:19:48.460 INFO:teuthology.orchestra.run.mira063.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-10-01T02:19:48.461 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-01T02:19:48.461 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-01T02:19:48.463 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.463 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.463 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.464 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.464 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:19:48.464 INFO:teuthology.orchestra.run.mira063.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-10-01T02:19:48.464 INFO:teuthology.orchestra.run.mira063.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.465 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.465 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.465 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-cephfs.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.465 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-10-01T02:19:48.467 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.467 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-rados = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.467 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-rados.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.467 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-rbd.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.468 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-rgw.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.468 INFO:teuthology.orchestra.run.mira063.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-10-01T02:19:48.468 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.469 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.470 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.470 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.473 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:19:48.473 INFO:teuthology.orchestra.run.mira064.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-10-01T02:19:48.473 INFO:teuthology.orchestra.run.mira064.stdout:---> Package libcephfs2.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.475 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-10-01T02:19:48.475 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librbd1.x86_64 2:15.0.0-5673.g2c0b216.el7 will be an update 2019-10-01T02:19:48.477 INFO:teuthology.orchestra.run.mira064.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-10-01T02:19:48.479 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-01T02:19:48.480 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-10-01T02:19:48.481 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:19:48.481 INFO:teuthology.orchestra.run.mira064.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-10-01T02:19:48.484 INFO:teuthology.orchestra.run.mira064.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.485 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.486 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.486 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-cephfs.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.489 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-10-01T02:19:48.490 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-rados.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.490 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-rbd.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.490 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-rgw.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:19:48.491 INFO:teuthology.orchestra.run.mira064.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-10-01T02:19:50.205 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:19:50.232 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:19:50.374 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:19:50.374 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:19:50.381 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:19:50.381 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout: ceph-radosgw x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 6.0 M 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:19:50.382 INFO:teuthology.orchestra.run.mira063.stdout: ceph-base x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 5.4 M 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: ceph-common x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 19 M 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: ceph-selinux x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 20 k 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: libcephfs2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 482 k 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-10-01T02:19:50.383 INFO:teuthology.orchestra.run.mira063.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: libradosstriper1 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 360 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: librgw2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 5.3 M 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-argparse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 35 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-common x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 13 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: python-cephfs x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 95 k 2019-10-01T02:19:50.384 INFO:teuthology.orchestra.run.mira063.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-10-01T02:19:50.385 INFO:teuthology.orchestra.run.mira063.stdout: python-rados x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 208 k 2019-10-01T02:19:50.385 INFO:teuthology.orchestra.run.mira063.stdout: python-rbd x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 188 k 2019-10-01T02:19:50.385 INFO:teuthology.orchestra.run.mira063.stdout: python-rgw x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 77 k 2019-10-01T02:19:50.385 INFO:teuthology.orchestra.run.mira063.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-10-01T02:19:50.385 INFO:teuthology.orchestra.run.mira063.stdout:Updating for dependencies: 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout: librados2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 3.4 M 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout: librbd1 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.7 M 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+19 Dependent packages) 2019-10-01T02:19:50.386 INFO:teuthology.orchestra.run.mira063.stdout:Upgrade ( 2 Dependent packages) 2019-10-01T02:19:50.387 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:19:50.387 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 43 M 2019-10-01T02:19:50.387 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:19:50.388 INFO:teuthology.orchestra.run.mira063.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-01T02:19:50.401 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:19:50.402 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:19:50.412 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:19:50.412 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:19:50.413 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:19:50.414 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:19:50.415 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:19:50.416 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:19:50.417 INFO:teuthology.orchestra.run.mira064.stdout: ceph-radosgw x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 6.0 M 2019-10-01T02:19:50.417 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:19:50.417 INFO:teuthology.orchestra.run.mira064.stdout: ceph-base x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 5.4 M 2019-10-01T02:19:50.418 INFO:teuthology.orchestra.run.mira064.stdout: ceph-common x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 19 M 2019-10-01T02:19:50.418 INFO:teuthology.orchestra.run.mira064.stdout: ceph-selinux x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 20 k 2019-10-01T02:19:50.418 INFO:teuthology.orchestra.run.mira064.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-10-01T02:19:50.418 INFO:teuthology.orchestra.run.mira064.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-10-01T02:19:50.419 INFO:teuthology.orchestra.run.mira064.stdout: libcephfs2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 482 k 2019-10-01T02:19:50.419 INFO:teuthology.orchestra.run.mira064.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-10-01T02:19:50.420 INFO:teuthology.orchestra.run.mira064.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-10-01T02:19:50.420 INFO:teuthology.orchestra.run.mira064.stdout: libradosstriper1 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 360 k 2019-10-01T02:19:50.420 INFO:teuthology.orchestra.run.mira064.stdout: librgw2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 5.3 M 2019-10-01T02:19:50.420 INFO:teuthology.orchestra.run.mira064.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-10-01T02:19:50.420 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-argparse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 35 k 2019-10-01T02:19:50.421 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-common x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 13 k 2019-10-01T02:19:50.429 INFO:teuthology.orchestra.run.mira064.stdout: python-cephfs x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 95 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: python-rados x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 208 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: python-rbd x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 188 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: python-rgw x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 77 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout:Updating for dependencies: 2019-10-01T02:19:50.430 INFO:teuthology.orchestra.run.mira064.stdout: librados2 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 3.4 M 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout: librbd1 x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.7 M 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+19 Dependent packages) 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout:Upgrade ( 2 Dependent packages) 2019-10-01T02:19:50.431 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:19:50.432 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 43 M 2019-10-01T02:19:50.432 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:19:50.432 INFO:teuthology.orchestra.run.mira064.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-10-01T02:20:27.009 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:20:27.009 INFO:teuthology.orchestra.run.mira063.stdout:Total 1.2 MB/s | 43 MB 00:36 2019-10-01T02:20:27.031 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:20:27.136 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:20:27.402 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:20:27.403 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:20:27.466 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:20:27.467 INFO:teuthology.orchestra.run.mira064.stdout:Total 1.2 MB/s | 43 MB 00:37 2019-10-01T02:20:27.485 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:20:27.581 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:20:27.655 INFO:teuthology.orchestra.run.mira063.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/24 2019-10-01T02:20:27.700 INFO:teuthology.orchestra.run.mira063.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/24 2019-10-01T02:20:27.735 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-ceph-argparse-15.0.0-5673.g2c0b216.el7.x86_64 3/24 2019-10-01T02:20:27.787 INFO:teuthology.orchestra.run.mira063.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/24 2019-10-01T02:20:27.853 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 5/24 2019-10-01T02:20:28.064 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:20:28.065 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:20:28.311 INFO:teuthology.orchestra.run.mira063.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/24 2019-10-01T02:20:28.319 INFO:teuthology.orchestra.run.mira064.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/24 2019-10-01T02:20:28.344 INFO:teuthology.orchestra.run.mira064.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/24 2019-10-01T02:20:28.375 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-ceph-argparse-15.0.0-5673.g2c0b216.el7.x86_64 3/24 2019-10-01T02:20:28.423 INFO:teuthology.orchestra.run.mira064.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/24 2019-10-01T02:20:28.451 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 5/24 2019-10-01T02:20:28.547 INFO:teuthology.orchestra.run.mira064.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/24 2019-10-01T02:20:28.849 INFO:teuthology.orchestra.run.mira063.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/24 2019-10-01T02:20:28.918 INFO:teuthology.orchestra.run.mira063.stdout: Updating : 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 8/24 2019-10-01T02:20:29.066 INFO:teuthology.orchestra.run.mira064.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/24 2019-10-01T02:20:29.144 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-rados-15.0.0-5673.g2c0b216.el7.x86_64 9/24 2019-10-01T02:20:29.433 INFO:teuthology.orchestra.run.mira064.stdout: Updating : 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 8/24 2019-10-01T02:20:29.673 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-rados-15.0.0-5673.g2c0b216.el7.x86_64 9/24 2019-10-01T02:20:29.885 INFO:teuthology.orchestra.run.mira063.stdout: Updating : 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 10/24 2019-10-01T02:20:29.975 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:librgw2-15.0.0-5673.g2c0b216.el7.x86_64 11/24 2019-10-01T02:20:30.019 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 12/24 2019-10-01T02:20:30.035 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-cephfs-15.0.0-5673.g2c0b216.el7.x86_64 13/24 2019-10-01T02:20:30.069 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-rgw-15.0.0-5673.g2c0b216.el7.x86_64 14/24 2019-10-01T02:20:30.120 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-rbd-15.0.0-5673.g2c0b216.el7.x86_64 15/24 2019-10-01T02:20:30.157 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:libradosstriper1-15.0.0-5673.g2c0b216.el7.x86_64 16/24 2019-10-01T02:20:30.185 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 17/24 2019-10-01T02:20:30.409 INFO:teuthology.orchestra.run.mira064.stdout: Updating : 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 10/24 2019-10-01T02:20:30.520 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:librgw2-15.0.0-5673.g2c0b216.el7.x86_64 11/24 2019-10-01T02:20:30.562 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 12/24 2019-10-01T02:20:30.579 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-cephfs-15.0.0-5673.g2c0b216.el7.x86_64 13/24 2019-10-01T02:20:30.612 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-rgw-15.0.0-5673.g2c0b216.el7.x86_64 14/24 2019-10-01T02:20:30.703 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-rbd-15.0.0-5673.g2c0b216.el7.x86_64 15/24 2019-10-01T02:20:30.745 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:libradosstriper1-15.0.0-5673.g2c0b216.el7.x86_64 16/24 2019-10-01T02:20:30.779 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 17/24 2019-10-01T02:20:32.772 INFO:teuthology.orchestra.run.mira063.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 18/24 2019-10-01T02:20:33.345 INFO:teuthology.orchestra.run.mira064.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 18/24 2019-10-01T02:20:33.751 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 19/24 2019-10-01T02:20:33.978 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 20/24 2019-10-01T02:20:34.317 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 19/24 2019-10-01T02:20:34.525 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 20/24 2019-10-01T02:20:50.559 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-selinux-15.0.0-5673.g2c0b216.el7.x86_64 21/24 2019-10-01T02:20:50.713 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 22/24 2019-10-01T02:20:50.756 INFO:teuthology.orchestra.run.mira063.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 23/24 2019-10-01T02:20:50.833 INFO:teuthology.orchestra.run.mira063.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 24/24 2019-10-01T02:20:50.841 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 1/24 2019-10-01T02:20:50.847 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 2/24 2019-10-01T02:20:50.853 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 3/24 2019-10-01T02:20:50.860 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 4/24 2019-10-01T02:20:50.865 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-rbd-15.0.0-5673.g2c0b216.el7.x86_64 5/24 2019-10-01T02:20:50.871 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-rados-15.0.0-5673.g2c0b216.el7.x86_64 6/24 2019-10-01T02:20:50.875 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 7/24 2019-10-01T02:20:50.880 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/24 2019-10-01T02:20:50.886 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-selinux-15.0.0-5673.g2c0b216.el7.x86_64 9/24 2019-10-01T02:20:50.891 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-cephfs-15.0.0-5673.g2c0b216.el7.x86_64 10/24 2019-10-01T02:20:50.893 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-selinux-15.0.0-5673.g2c0b216.el7.x86_64 21/24 2019-10-01T02:20:50.896 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 11/24 2019-10-01T02:20:50.901 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 12/24 2019-10-01T02:20:50.906 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:libradosstriper1-15.0.0-5673.g2c0b216.el7.x86_64 13/24 2019-10-01T02:20:50.913 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-ceph-argparse-15.0.0-5673.g2c0b216.el7.x86_64 14/24 2019-10-01T02:20:50.915 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:librgw2-15.0.0-5673.g2c0b216.el7.x86_64 15/24 2019-10-01T02:20:50.919 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 16/24 2019-10-01T02:20:50.924 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 17/24 2019-10-01T02:20:50.933 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 18/24 2019-10-01T02:20:50.933 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-rgw-15.0.0-5673.g2c0b216.el7.x86_64 19/24 2019-10-01T02:20:50.937 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 20/24 2019-10-01T02:20:50.942 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 21/24 2019-10-01T02:20:50.946 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 22/24 2019-10-01T02:20:50.947 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 23/24 2019-10-01T02:20:51.052 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 22/24 2019-10-01T02:20:51.090 INFO:teuthology.orchestra.run.mira064.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 23/24 2019-10-01T02:20:51.148 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 24/24 2019-10-01T02:20:51.148 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout: ceph-radosgw.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout: ceph-base.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout: ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.149 INFO:teuthology.orchestra.run.mira063.stdout: ceph-selinux.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: libcephfs2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: libradosstriper1.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: librgw2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.150 INFO:teuthology.orchestra.run.mira063.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-argparse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-cephfs.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-rados.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-rbd.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.151 INFO:teuthology.orchestra.run.mira063.stdout: python-rgw.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Updated: 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout: librados2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout: librbd1.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:51.152 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:20:51.160 INFO:teuthology.orchestra.run.mira064.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 24/24 2019-10-01T02:20:51.163 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 1/24 2019-10-01T02:20:51.168 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 2/24 2019-10-01T02:20:51.173 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 3/24 2019-10-01T02:20:51.177 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 4/24 2019-10-01T02:20:51.182 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-rbd-15.0.0-5673.g2c0b216.el7.x86_64 5/24 2019-10-01T02:20:51.186 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-rados-15.0.0-5673.g2c0b216.el7.x86_64 6/24 2019-10-01T02:20:51.190 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-base-15.0.0-5673.g2c0b216.el7.x86_64 7/24 2019-10-01T02:20:51.194 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/24 2019-10-01T02:20:51.199 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-selinux-15.0.0-5673.g2c0b216.el7.x86_64 9/24 2019-10-01T02:20:51.203 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-cephfs-15.0.0-5673.g2c0b216.el7.x86_64 10/24 2019-10-01T02:20:51.207 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 11/24 2019-10-01T02:20:51.211 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 12/24 2019-10-01T02:20:51.216 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:libradosstriper1-15.0.0-5673.g2c0b216.el7.x86_64 13/24 2019-10-01T02:20:51.220 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-ceph-argparse-15.0.0-5673.g2c0b216.el7.x86_64 14/24 2019-10-01T02:20:51.224 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:librgw2-15.0.0-5673.g2c0b216.el7.x86_64 15/24 2019-10-01T02:20:51.228 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 16/24 2019-10-01T02:20:51.233 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 17/24 2019-10-01T02:20:51.237 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-ceph-common-15.0.0-5673.g2c0b216.el7.x86_64 18/24 2019-10-01T02:20:51.242 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-rgw-15.0.0-5673.g2c0b216.el7.x86_64 19/24 2019-10-01T02:20:51.246 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 20/24 2019-10-01T02:20:51.250 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 21/24 2019-10-01T02:20:51.255 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-radosgw-15.0.0-5673.g2c0b216.el7.x86_64 22/24 2019-10-01T02:20:51.256 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:20:51.256 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-test 2019-10-01T02:20:51.256 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 23/24 2019-10-01T02:20:51.450 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 24/24 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout: ceph-radosgw.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout: ceph-base.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.451 INFO:teuthology.orchestra.run.mira064.stdout: ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: ceph-selinux.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: libcephfs2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-10-01T02:20:51.452 INFO:teuthology.orchestra.run.mira064.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: libradosstriper1.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: librgw2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-argparse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-common.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: python-cephfs.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-10-01T02:20:51.453 INFO:teuthology.orchestra.run.mira064.stdout: python-rados.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: python-rbd.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: python-rgw.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Updated: 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: librados2.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.454 INFO:teuthology.orchestra.run.mira064.stdout: librbd1.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:20:51.455 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:51.455 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:20:51.562 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:20:51.562 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-test 2019-10-01T02:20:51.669 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:20:51.745 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:20:51.747 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:20:51.748 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:20:51.748 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:20:51.749 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:20:51.927 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:20:52.003 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:20:52.005 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:20:52.006 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:20:52.007 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:20:52.007 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:20:53.178 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:20:53.439 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:20:53.791 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:20:53.793 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:20:53.793 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-test.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:20:53.828 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:53.883 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:53.884 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:53.887 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:20:53.887 INFO:teuthology.orchestra.run.mira063.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-10-01T02:20:53.897 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-10-01T02:20:53.897 INFO:teuthology.orchestra.run.mira063.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-10-01T02:20:53.897 INFO:teuthology.orchestra.run.mira063.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-10-01T02:20:53.903 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:20:53.904 INFO:teuthology.orchestra.run.mira063.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-10-01T02:20:54.052 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:20:54.053 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:20:54.053 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-test.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:20:54.088 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:54.156 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:54.156 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:20:54.156 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:20:54.156 INFO:teuthology.orchestra.run.mira064.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-10-01T02:20:54.157 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-10-01T02:20:54.165 INFO:teuthology.orchestra.run.mira064.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-10-01T02:20:54.165 INFO:teuthology.orchestra.run.mira064.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-10-01T02:20:54.168 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:20:54.168 INFO:teuthology.orchestra.run.mira064.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-10-01T02:20:54.645 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:20:54.909 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:20:55.071 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:55.072 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:20:55.074 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:55.074 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:20:55.074 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:20:55.074 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:20:55.074 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout: ceph-test x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 36 M 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-10-01T02:20:55.075 INFO:teuthology.orchestra.run.mira063.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-10-01T02:20:55.076 INFO:teuthology.orchestra.run.mira063.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-10-01T02:20:55.076 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:55.076 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:20:55.076 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:20:55.077 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:20:55.077 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:20:55.078 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 37 M 2019-10-01T02:20:55.078 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 130 M 2019-10-01T02:20:55.078 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:20:55.333 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:55.333 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:20:55.335 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:20:55.336 INFO:teuthology.orchestra.run.mira064.stdout: ceph-test x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 36 M 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:55.337 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:20:55.338 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:20:55.338 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:20:55.338 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:20:55.338 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 37 M 2019-10-01T02:20:55.338 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 130 M 2019-10-01T02:20:55.339 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:21:18.482 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:18.482 INFO:teuthology.orchestra.run.mira064.stdout:Total 1.6 MB/s | 37 MB 00:23 2019-10-01T02:21:18.486 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:21:18.491 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:18.491 INFO:teuthology.orchestra.run.mira063.stdout:Total 1.6 MB/s | 37 MB 00:23 2019-10-01T02:21:18.495 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:21:18.516 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:21:18.526 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:21:18.596 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:21:18.596 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:21:18.605 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:21:18.606 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:21:18.880 INFO:teuthology.orchestra.run.mira063.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-10-01T02:21:18.884 INFO:teuthology.orchestra.run.mira064.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-10-01T02:21:18.908 INFO:teuthology.orchestra.run.mira063.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-10-01T02:21:18.915 INFO:teuthology.orchestra.run.mira064.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-10-01T02:21:18.970 INFO:teuthology.orchestra.run.mira064.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-10-01T02:21:18.981 INFO:teuthology.orchestra.run.mira063.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-10-01T02:21:24.087 INFO:teuthology.orchestra.run.mira064.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-10-01T02:21:24.104 INFO:teuthology.orchestra.run.mira063.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-10-01T02:21:24.108 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 5/5 2019-10-01T02:21:24.114 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 1/5 2019-10-01T02:21:24.118 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2/5 2019-10-01T02:21:24.125 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 5/5 2019-10-01T02:21:24.127 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-10-01T02:21:24.131 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 1/5 2019-10-01T02:21:24.132 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-10-01T02:21:24.140 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-test-15.0.0-5673.g2c0b216.el7.x86_64 2/5 2019-10-01T02:21:24.144 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-10-01T02:21:24.148 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-10-01T02:21:24.321 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-10-01T02:21:24.321 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:24.321 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:21:24.322 INFO:teuthology.orchestra.run.mira064.stdout: ceph-test.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:24.322 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:24.322 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:21:24.322 INFO:teuthology.orchestra.run.mira064.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-10-01T02:21:24.323 INFO:teuthology.orchestra.run.mira064.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-10-01T02:21:24.323 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:24.323 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:21:24.336 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-10-01T02:21:24.336 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:24.336 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:21:24.336 INFO:teuthology.orchestra.run.mira063.stdout: ceph-test.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:24.337 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:24.337 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:21:24.337 INFO:teuthology.orchestra.run.mira063.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-10-01T02:21:24.337 INFO:teuthology.orchestra.run.mira063.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-10-01T02:21:24.337 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:24.338 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:21:24.430 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:21:24.430 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph 2019-10-01T02:21:24.447 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:21:24.447 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph 2019-10-01T02:21:24.750 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:24.797 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:24.830 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:24.830 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:24.830 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:24.831 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:24.831 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:24.873 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:24.875 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:24.876 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:24.876 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:24.877 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:26.288 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:26.301 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:26.901 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:21:26.902 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:26.902 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:26.907 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.913 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:21:26.914 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:26.914 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:26.919 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.962 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.964 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.967 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.970 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:26.970 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mds.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:26.974 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.977 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.979 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.982 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:26.982 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mds.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:26.989 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:26.994 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:26.996 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.004 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:27.005 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.005 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.005 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.005 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.006 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mon.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:27.006 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.008 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.010 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.010 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-osd.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:27.012 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.014 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.016 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:21:27.018 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mon.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:27.019 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:27.019 INFO:teuthology.orchestra.run.mira063.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-10-01T02:21:27.021 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-10-01T02:21:27.023 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-10-01T02:21:27.023 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-osd.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:27.024 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.026 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.027 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.029 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.031 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.032 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:27.032 INFO:teuthology.orchestra.run.mira064.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-10-01T02:21:27.033 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-requests.noarch 0:2.6.0-5.el7 will be installed 2019-10-01T02:21:27.034 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-10-01T02:21:27.034 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-5.el7.noarch 2019-10-01T02:21:27.035 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-10-01T02:21:27.036 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-10-01T02:21:27.036 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-10-01T02:21:27.037 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-10-01T02:21:27.037 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.038 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:27.039 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.039 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-10-01T02:21:27.039 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-10-01T02:21:27.043 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-10-01T02:21:27.043 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-10-01T02:21:27.043 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-10-01T02:21:27.044 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-10-01T02:21:27.044 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.044 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.045 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-10-01T02:21:27.046 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-10-01T02:21:27.046 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-10-01T02:21:27.046 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:27.046 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-10-01T02:21:27.046 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-10-01T02:21:27.047 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-10-01T02:21:27.047 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-requests.noarch 0:2.6.0-5.el7 will be installed 2019-10-01T02:21:27.048 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-10-01T02:21:27.048 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:27.048 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-10-01T02:21:27.048 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-10-01T02:21:27.049 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-5.el7.noarch 2019-10-01T02:21:27.050 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:27.050 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-10-01T02:21:27.050 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-10-01T02:21:27.050 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-10-01T02:21:27.050 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-10-01T02:21:27.051 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:27.051 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-10-01T02:21:27.052 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-10-01T02:21:27.054 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-10-01T02:21:27.055 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-10-01T02:21:27.055 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-10-01T02:21:27.055 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-10-01T02:21:27.056 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-10-01T02:21:27.057 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-10-01T02:21:27.057 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-10-01T02:21:27.058 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:27.058 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-10-01T02:21:27.058 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-10-01T02:21:27.060 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-10-01T02:21:27.060 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:27.060 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-10-01T02:21:27.060 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-10-01T02:21:27.062 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:27.062 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-10-01T02:21:27.786 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:27.801 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:28.216 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:28.216 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:21:28.223 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:28.223 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:28.223 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:21:28.223 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:21:28.223 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout: ceph x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 2.0 k 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mds x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.9 M 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.6 M 2019-10-01T02:21:28.224 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mon x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 4.3 M 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: ceph-osd x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 16 M 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-10-01T02:21:28.225 INFO:teuthology.orchestra.run.mira063.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-10-01T02:21:28.226 INFO:teuthology.orchestra.run.mira063.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-10-01T02:21:28.227 INFO:teuthology.orchestra.run.mira063.stdout: python-requests noarch 2.6.0-5.el7 base 94 k 2019-10-01T02:21:28.227 INFO:teuthology.orchestra.run.mira063.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-10-01T02:21:28.227 INFO:teuthology.orchestra.run.mira063.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-10-01T02:21:28.227 INFO:teuthology.orchestra.run.mira063.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-10-01T02:21:28.227 INFO:teuthology.orchestra.run.mira063.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:28.228 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:21:28.229 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:28.229 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+21 Dependent packages) 2019-10-01T02:21:28.229 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:28.229 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 27 M 2019-10-01T02:21:28.230 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 86 M 2019-10-01T02:21:28.230 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:21:28.232 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:28.232 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:21:28.239 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:28.239 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:28.239 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:21:28.239 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout: ceph x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 2.0 k 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mds x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.9 M 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 1.6 M 2019-10-01T02:21:28.240 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mon x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 4.3 M 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: ceph-osd x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 16 M 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-10-01T02:21:28.241 INFO:teuthology.orchestra.run.mira064.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-requests noarch 2.6.0-5.el7 base 94 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-10-01T02:21:28.242 INFO:teuthology.orchestra.run.mira064.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:21:28.243 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:28.244 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+21 Dependent packages) 2019-10-01T02:21:28.244 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:28.244 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 27 M 2019-10-01T02:21:28.244 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 86 M 2019-10-01T02:21:28.245 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:21:30.194 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:30.194 INFO:teuthology.orchestra.run.mira063.stdout:Total 14 MB/s | 27 MB 00:01 2019-10-01T02:21:30.220 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:21:30.273 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:21:30.454 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:30.454 INFO:teuthology.orchestra.run.mira064.stdout:Total 12 MB/s | 27 MB 00:02 2019-10-01T02:21:30.475 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:21:30.515 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:21:30.605 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:21:30.606 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:21:30.899 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:21:30.900 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:21:30.901 INFO:teuthology.orchestra.run.mira063.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 1/22 2019-10-01T02:21:30.947 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-webob-1.2.3-7.el7.noarch 2/22 2019-10-01T02:21:31.192 INFO:teuthology.orchestra.run.mira064.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 1/22 2019-10-01T02:21:31.237 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-webob-1.2.3-7.el7.noarch 2/22 2019-10-01T02:21:31.593 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-10-01T02:21:31.797 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-10-01T02:21:32.145 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mon-15.0.0-5673.g2c0b216.el7.x86_64 4/22 2019-10-01T02:21:32.262 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mds-15.0.0-5673.g2c0b216.el7.x86_64 5/22 2019-10-01T02:21:32.404 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mon-15.0.0-5673.g2c0b216.el7.x86_64 4/22 2019-10-01T02:21:32.523 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mds-15.0.0-5673.g2c0b216.el7.x86_64 5/22 2019-10-01T02:21:32.546 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 6/22 2019-10-01T02:21:32.595 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 7/22 2019-10-01T02:21:32.736 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 8/22 2019-10-01T02:21:32.756 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/22 2019-10-01T02:21:32.805 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 6/22 2019-10-01T02:21:32.855 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 7/22 2019-10-01T02:21:32.869 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 10/22 2019-10-01T02:21:32.887 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-mako-0.8.1-2.el7.noarch 11/22 2019-10-01T02:21:32.941 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 12/22 2019-10-01T02:21:32.980 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 13/22 2019-10-01T02:21:32.994 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 8/22 2019-10-01T02:21:33.014 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/22 2019-10-01T02:21:33.138 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 10/22 2019-10-01T02:21:33.153 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-mako-0.8.1-2.el7.noarch 11/22 2019-10-01T02:21:33.206 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 12/22 2019-10-01T02:21:33.246 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 13/22 2019-10-01T02:21:35.075 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-requests-2.6.0-5.el7.noarch 14/22 2019-10-01T02:21:35.343 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-requests-2.6.0-5.el7.noarch 14/22 2019-10-01T02:21:35.374 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-osd-15.0.0-5673.g2c0b216.el7.x86_64 15/22 2019-10-01T02:21:35.382 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 16/22 2019-10-01T02:21:35.398 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-10-01T02:21:35.408 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-10-01T02:21:35.555 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-10-01T02:21:35.698 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-osd-15.0.0-5673.g2c0b216.el7.x86_64 15/22 2019-10-01T02:21:35.712 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 16/22 2019-10-01T02:21:35.741 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-10-01T02:21:35.758 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-10-01T02:21:35.946 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-10-01T02:21:35.989 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-10-01T02:21:36.111 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 21/22 2019-10-01T02:21:36.153 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 22/22 2019-10-01T02:21:36.162 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 1/22 2019-10-01T02:21:36.170 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 2/22 2019-10-01T02:21:36.178 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 3/22 2019-10-01T02:21:36.185 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 4/22 2019-10-01T02:21:36.192 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-osd-15.0.0-5673.g2c0b216.el7.x86_64 5/22 2019-10-01T02:21:36.199 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 6/22 2019-10-01T02:21:36.205 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 7/22 2019-10-01T02:21:36.212 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/22 2019-10-01T02:21:36.217 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 9/22 2019-10-01T02:21:36.223 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 10/22 2019-10-01T02:21:36.229 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-10-01T02:21:36.235 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 12/22 2019-10-01T02:21:36.240 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 13/22 2019-10-01T02:21:36.246 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 14/22 2019-10-01T02:21:36.252 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 15/22 2019-10-01T02:21:36.257 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 16/22 2019-10-01T02:21:36.262 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-requests-2.6.0-5.el7.noarch 17/22 2019-10-01T02:21:36.267 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 18/22 2019-10-01T02:21:36.272 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 19/22 2019-10-01T02:21:36.277 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 20/22 2019-10-01T02:21:36.282 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mds-15.0.0-5673.g2c0b216.el7.x86_64 21/22 2019-10-01T02:21:36.287 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-10-01T02:21:36.396 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 21/22 2019-10-01T02:21:36.443 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 22/22 2019-10-01T02:21:36.457 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 1/22 2019-10-01T02:21:36.460 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 2/22 2019-10-01T02:21:36.466 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 3/22 2019-10-01T02:21:36.472 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 4/22 2019-10-01T02:21:36.475 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mon-15.0.0-5673.g2c0b216.el7.x86_64 22/22 2019-10-01T02:21:36.476 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:36.476 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:21:36.476 INFO:teuthology.orchestra.run.mira063.stdout: ceph.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.476 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:36.477 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:21:36.477 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mds.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.477 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.477 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mon.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.478 INFO:teuthology.orchestra.run.mira063.stdout: ceph-osd.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.478 INFO:teuthology.orchestra.run.mira063.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-10-01T02:21:36.478 INFO:teuthology.orchestra.run.mira063.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-10-01T02:21:36.478 INFO:teuthology.orchestra.run.mira063.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-10-01T02:21:36.478 INFO:teuthology.orchestra.run.mira063.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-10-01T02:21:36.479 INFO:teuthology.orchestra.run.mira063.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-10-01T02:21:36.479 INFO:teuthology.orchestra.run.mira063.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-10-01T02:21:36.479 INFO:teuthology.orchestra.run.mira063.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-10-01T02:21:36.479 INFO:teuthology.orchestra.run.mira063.stdout: python-requests.noarch 0:2.6.0-5.el7 2019-10-01T02:21:36.480 INFO:teuthology.orchestra.run.mira063.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-10-01T02:21:36.480 INFO:teuthology.orchestra.run.mira063.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-10-01T02:21:36.480 INFO:teuthology.orchestra.run.mira063.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-10-01T02:21:36.480 INFO:teuthology.orchestra.run.mira063.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-10-01T02:21:36.481 INFO:teuthology.orchestra.run.mira063.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-10-01T02:21:36.481 INFO:teuthology.orchestra.run.mira063.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-10-01T02:21:36.481 INFO:teuthology.orchestra.run.mira063.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-10-01T02:21:36.481 INFO:teuthology.orchestra.run.mira063.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-10-01T02:21:36.482 INFO:teuthology.orchestra.run.mira063.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-10-01T02:21:36.482 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:36.483 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:21:36.484 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-osd-15.0.0-5673.g2c0b216.el7.x86_64 5/22 2019-10-01T02:21:36.484 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 6/22 2019-10-01T02:21:36.489 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 7/22 2019-10-01T02:21:36.494 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/22 2019-10-01T02:21:36.500 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 9/22 2019-10-01T02:21:36.506 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 10/22 2019-10-01T02:21:36.511 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-10-01T02:21:36.516 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 12/22 2019-10-01T02:21:36.522 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 13/22 2019-10-01T02:21:36.537 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 14/22 2019-10-01T02:21:36.537 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 15/22 2019-10-01T02:21:36.538 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 16/22 2019-10-01T02:21:36.543 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-requests-2.6.0-5.el7.noarch 17/22 2019-10-01T02:21:36.546 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-15.0.0-5673.g2c0b216.el7.x86_64 18/22 2019-10-01T02:21:36.550 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 19/22 2019-10-01T02:21:36.555 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 20/22 2019-10-01T02:21:36.559 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mds-15.0.0-5673.g2c0b216.el7.x86_64 21/22 2019-10-01T02:21:36.584 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:21:36.584 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr 2019-10-01T02:21:36.750 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mon-15.0.0-5673.g2c0b216.el7.x86_64 22/22 2019-10-01T02:21:36.750 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:36.750 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout: ceph.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mds.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.751 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mon.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: ceph-osd.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-10-01T02:21:36.752 INFO:teuthology.orchestra.run.mira064.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-requests.noarch 0:2.6.0-5.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-10-01T02:21:36.753 INFO:teuthology.orchestra.run.mira064.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-10-01T02:21:36.754 INFO:teuthology.orchestra.run.mira064.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-10-01T02:21:36.754 INFO:teuthology.orchestra.run.mira064.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-10-01T02:21:36.754 INFO:teuthology.orchestra.run.mira064.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-10-01T02:21:36.754 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:36.754 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:21:36.861 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:21:36.862 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr 2019-10-01T02:21:36.976 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:37.053 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:37.055 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:37.056 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:37.056 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:37.057 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:37.252 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:37.329 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:37.331 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:37.331 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:37.332 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:37.332 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:38.495 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:38.768 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:39.110 INFO:teuthology.orchestra.run.mira063.stdout:Package 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:21:39.111 INFO:teuthology.orchestra.run.mira063.stdout:Nothing to do 2019-10-01T02:21:39.207 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:21:39.207 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr-dashboard 2019-10-01T02:21:39.385 INFO:teuthology.orchestra.run.mira064.stdout:Package 2:ceph-mgr-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:21:39.385 INFO:teuthology.orchestra.run.mira064.stdout:Nothing to do 2019-10-01T02:21:39.481 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:21:39.481 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr-dashboard 2019-10-01T02:21:39.578 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:39.654 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:39.656 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:39.657 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:39.657 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:39.664 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:39.830 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:39.907 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:39.908 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:39.909 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:39.909 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:39.910 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:41.088 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:41.337 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:41.703 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:21:41.703 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:41.704 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:41.710 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:41.765 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:41.767 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:41.767 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:41.767 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-10-01T02:21:41.768 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-10-01T02:21:41.770 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:41.770 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-10-01T02:21:41.955 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:21:41.969 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:41.970 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:41.970 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-5673.g2c0b216.el7 for package: 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:42.021 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:42.022 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:42.022 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:42.022 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-10-01T02:21:42.022 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-10-01T02:21:42.024 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:42.024 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-10-01T02:21:42.526 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:42.775 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:42.970 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:42.970 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:21:42.972 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-dashboard noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 4.6 M 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:21:42.973 INFO:teuthology.orchestra.run.mira063.stdout: ceph-grafana-dashboards noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 18 k 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:42.974 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+3 Dependent packages) 2019-10-01T02:21:42.975 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:42.975 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 5.3 M 2019-10-01T02:21:42.975 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 27 M 2019-10-01T02:21:42.975 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:21:43.212 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:43.212 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:21:43.214 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:43.214 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:43.214 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:21:43.214 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:43.215 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:21:43.215 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-dashboard noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 4.6 M 2019-10-01T02:21:43.215 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:21:43.215 INFO:teuthology.orchestra.run.mira064.stdout: ceph-grafana-dashboards noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 18 k 2019-10-01T02:21:43.216 INFO:teuthology.orchestra.run.mira064.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-10-01T02:21:43.216 INFO:teuthology.orchestra.run.mira064.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-10-01T02:21:43.216 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:43.216 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:21:43.216 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:43.217 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+3 Dependent packages) 2019-10-01T02:21:43.217 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:43.218 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 5.3 M 2019-10-01T02:21:43.218 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 27 M 2019-10-01T02:21:43.218 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:21:47.205 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:47.205 INFO:teuthology.orchestra.run.mira063.stdout:Total 1.3 MB/s | 5.3 MB 00:04 2019-10-01T02:21:47.213 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:47.213 INFO:teuthology.orchestra.run.mira064.stdout:Total 1.3 MB/s | 5.3 MB 00:03 2019-10-01T02:21:47.213 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:21:47.216 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:21:47.226 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:21:47.230 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:21:47.433 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:21:47.433 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:21:47.507 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:21:47.507 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:21:47.764 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-10-01T02:21:47.780 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-routes-1.13-2.el7.noarch 2/4 2019-10-01T02:21:47.865 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-10-01T02:21:47.881 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-routes-1.13-2.el7.noarch 2/4 2019-10-01T02:21:49.104 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-5673.g2c0b216.el7.noarch 3/4 2019-10-01T02:21:49.155 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 4/4 2019-10-01T02:21:49.160 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 1/4 2019-10-01T02:21:49.165 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-5673.g2c0b216.el7.noarch 2/4 2019-10-01T02:21:49.170 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-routes-1.13-2.el7.noarch 3/4 2019-10-01T02:21:49.213 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-5673.g2c0b216.el7.noarch 3/4 2019-10-01T02:21:49.270 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 4/4 2019-10-01T02:21:49.275 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-5673.g2c0b216.el7.noarch 1/4 2019-10-01T02:21:49.279 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-5673.g2c0b216.el7.noarch 2/4 2019-10-01T02:21:49.285 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-routes-1.13-2.el7.noarch 3/4 2019-10-01T02:21:49.367 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 4/4 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:21:49.368 INFO:teuthology.orchestra.run.mira063.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:49.369 INFO:teuthology.orchestra.run.mira063.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-10-01T02:21:49.369 INFO:teuthology.orchestra.run.mira063.stdout: python-routes.noarch 0:1.13-2.el7 2019-10-01T02:21:49.369 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:49.369 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:21:49.478 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 4/4 2019-10-01T02:21:49.478 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:49.478 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:21:49.478 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:49.478 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout: python-routes.noarch 0:1.13-2.el7 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:49.479 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:21:49.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:21:49.480 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-10-01T02:21:49.589 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:21:49.590 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-10-01T02:21:49.783 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:49.859 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:49.861 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:49.862 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:49.862 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:49.863 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:49.864 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:49.940 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:49.942 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:49.943 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:49.943 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:49.944 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:51.303 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:51.385 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:51.919 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:21:51.920 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:51.920 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:51.999 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:21:52.000 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:52.000 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:52.673 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:52.757 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:53.047 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:53.047 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:21:53.115 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:53.116 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:53.116 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:21:53.116 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:53.116 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:21:53.116 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-diskprediction-cloud 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout: noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 83 k 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package 2019-10-01T02:21:53.117 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:53.118 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 83 k 2019-10-01T02:21:53.118 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 412 k 2019-10-01T02:21:53.118 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:21:53.128 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:53.128 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:21:53.197 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:53.197 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:53.197 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:21:53.197 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:53.198 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:21:53.198 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-diskprediction-cloud 2019-10-01T02:21:53.198 INFO:teuthology.orchestra.run.mira064.stdout: noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 83 k 2019-10-01T02:21:53.198 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:53.198 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 83 k 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 412 k 2019-10-01T02:21:53.199 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:21:53.541 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:21:53.543 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:21:53.594 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:21:53.595 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:21:53.625 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:21:53.628 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:21:53.679 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:21:53.679 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:21:53.843 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-5673.g2c0b216.el7.n 1/1 2019-10-01T02:21:53.932 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-5673.g2c0b216.el7.n 1/1 2019-10-01T02:21:54.037 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-5673.g2c0b216.el7.n 1/1 2019-10-01T02:21:54.037 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:54.037 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:21:54.037 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:54.038 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:54.038 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:21:54.134 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-5673.g2c0b216.el7.n 1/1 2019-10-01T02:21:54.134 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:54.135 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:21:54.135 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:21:54.135 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:54.135 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:21:54.143 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:21:54.143 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr-diskprediction-local 2019-10-01T02:21:54.242 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:21:54.242 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr-diskprediction-local 2019-10-01T02:21:54.425 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:54.502 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:54.504 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:54.504 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:54.505 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:54.505 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:54.595 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:21:54.670 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:21:54.672 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:21:54.673 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:21:54.673 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:21:54.674 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:21:55.946 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:56.105 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:21:56.573 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:21:56.573 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:56.574 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:56.578 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:56.634 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:56.634 INFO:teuthology.orchestra.run.mira063.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-10-01T02:21:56.647 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.650 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.653 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.654 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:56.654 INFO:teuthology.orchestra.run.mira063.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-10-01T02:21:56.656 INFO:teuthology.orchestra.run.mira063.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-10-01T02:21:56.658 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-01T02:21:56.661 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-01T02:21:56.662 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:21:56.662 INFO:teuthology.orchestra.run.mira063.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-10-01T02:21:56.719 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:21:56.720 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:56.720 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:21:56.724 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:21:56.780 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:56.780 INFO:teuthology.orchestra.run.mira064.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-10-01T02:21:56.793 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.796 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.799 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-10-01T02:21:56.800 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:56.800 INFO:teuthology.orchestra.run.mira064.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-10-01T02:21:56.802 INFO:teuthology.orchestra.run.mira064.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-10-01T02:21:56.804 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-01T02:21:56.807 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-10-01T02:21:56.807 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:21:56.807 INFO:teuthology.orchestra.run.mira064.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-10-01T02:21:57.407 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:57.561 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:21:57.836 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:57.837 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:21:57.837 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:57.837 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-diskprediction-local 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout: noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 1.1 M 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:21:57.838 INFO:teuthology.orchestra.run.mira063.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:21:57.839 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:21:57.840 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:21:57.840 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 12 M 2019-10-01T02:21:57.840 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 66 M 2019-10-01T02:21:57.844 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:21:57.994 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:57.994 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:21:57.996 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:57.996 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:57.996 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:21:57.996 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:57.996 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-diskprediction-local 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 1.1 M 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-10-01T02:21:57.997 INFO:teuthology.orchestra.run.mira064.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-10-01T02:21:57.998 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:57.998 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:21:57.998 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:21:57.998 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:21:57.998 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:21:57.999 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 12 M 2019-10-01T02:21:57.999 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 66 M 2019-10-01T02:21:57.999 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:21:58.367 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:58.367 INFO:teuthology.orchestra.run.mira063.stdout:Total 22 MB/s | 12 MB 00:00 2019-10-01T02:21:58.383 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:21:58.424 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:21:58.567 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:21:58.567 INFO:teuthology.orchestra.run.mira064.stdout:Total 20 MB/s | 12 MB 00:00 2019-10-01T02:21:58.582 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:21:58.625 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:21:58.660 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:21:58.660 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:21:58.833 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:21:58.833 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:21:59.063 INFO:teuthology.orchestra.run.mira063.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-10-01T02:21:59.176 INFO:teuthology.orchestra.run.mira063.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-10-01T02:21:59.238 INFO:teuthology.orchestra.run.mira064.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-10-01T02:21:59.351 INFO:teuthology.orchestra.run.mira064.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-10-01T02:22:01.210 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-10-01T02:22:01.386 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-10-01T02:22:01.897 INFO:teuthology.orchestra.run.mira063.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-10-01T02:22:02.031 INFO:teuthology.orchestra.run.mira064.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-10-01T02:22:02.175 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.n 5/5 2019-10-01T02:22:02.187 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-10-01T02:22:02.196 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.n 2/5 2019-10-01T02:22:02.206 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 3/5 2019-10-01T02:22:02.214 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 4/5 2019-10-01T02:22:02.268 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.n 5/5 2019-10-01T02:22:02.274 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-10-01T02:22:02.280 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-5673.g2c0b216.el7.n 2/5 2019-10-01T02:22:02.284 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 3/5 2019-10-01T02:22:02.289 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 4/5 2019-10-01T02:22:02.444 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-10-01T02:22:02.444 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:02.444 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:02.444 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:02.444 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:02.445 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:22:02.445 INFO:teuthology.orchestra.run.mira063.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-10-01T02:22:02.445 INFO:teuthology.orchestra.run.mira063.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-10-01T02:22:02.445 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:02.445 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:02.478 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:22:02.479 INFO:teuthology.orchestra.run.mira064.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-10-01T02:22:02.479 INFO:teuthology.orchestra.run.mira064.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-10-01T02:22:02.479 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:02.479 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:02.552 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:02.552 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr-rook 2019-10-01T02:22:02.589 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:02.589 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr-rook 2019-10-01T02:22:02.910 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:02.920 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:02.985 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:02.987 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:02.988 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:02.988 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:02.989 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:02.996 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:02.998 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:02.999 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:02.999 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:03.000 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:04.441 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:04.482 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:05.043 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:05.043 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:05.043 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:05.047 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:22:05.103 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:05.103 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-kubernetes.noarch 0:8.0.1-1.el7 will be installed 2019-10-01T02:22:05.108 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.110 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.112 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.113 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.115 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:05.116 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-10-01T02:22:05.116 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-10-01T02:22:05.117 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-10-01T02:22:05.118 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.120 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.121 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.123 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-10-01T02:22:05.124 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-10-01T02:22:05.125 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:05.126 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-10-01T02:22:05.126 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-10-01T02:22:05.127 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-10-01T02:22:05.129 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-10-01T02:22:05.129 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-10-01T02:22:05.130 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:05.130 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-10-01T02:22:05.133 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:05.134 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:05.134 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:05.137 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:22:05.193 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:05.193 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-kubernetes.noarch 0:8.0.1-1.el7 will be installed 2019-10-01T02:22:05.198 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.200 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.201 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.203 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-8.0.1-1.el7.noarch 2019-10-01T02:22:05.205 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:05.205 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-10-01T02:22:05.206 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-10-01T02:22:05.206 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-10-01T02:22:05.208 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.209 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.211 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-10-01T02:22:05.213 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-10-01T02:22:05.213 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-10-01T02:22:05.215 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:05.215 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-10-01T02:22:05.215 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-10-01T02:22:05.216 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-10-01T02:22:05.220 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-10-01T02:22:05.220 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-10-01T02:22:05.220 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:05.220 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-10-01T02:22:05.880 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:05.968 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:06.319 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:06.320 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:06.322 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:06.322 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:06.323 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:22:06.323 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:06.323 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:06.323 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-rook noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 24 k 2019-10-01T02:22:06.324 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:22:06.324 INFO:teuthology.orchestra.run.mira063.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-10-01T02:22:06.324 INFO:teuthology.orchestra.run.mira063.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-10-01T02:22:06.324 INFO:teuthology.orchestra.run.mira063.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-10-01T02:22:06.324 INFO:teuthology.orchestra.run.mira063.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-10-01T02:22:06.325 INFO:teuthology.orchestra.run.mira063.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-10-01T02:22:06.325 INFO:teuthology.orchestra.run.mira063.stdout: python2-kubernetes noarch 8.0.1-1.el7 epel 1.0 M 2019-10-01T02:22:06.325 INFO:teuthology.orchestra.run.mira063.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-10-01T02:22:06.325 INFO:teuthology.orchestra.run.mira063.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-10-01T02:22:06.326 INFO:teuthology.orchestra.run.mira063.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-10-01T02:22:06.326 INFO:teuthology.orchestra.run.mira063.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-10-01T02:22:06.326 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:06.326 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:06.327 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:06.327 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+10 Dependent packages) 2019-10-01T02:22:06.327 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:06.329 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 1.6 M 2019-10-01T02:22:06.330 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 22 M 2019-10-01T02:22:06.330 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:06.401 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:06.401 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:06.406 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:06.406 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:06.406 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:22:06.407 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:06.407 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:06.407 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-rook noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 24 k 2019-10-01T02:22:06.407 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:22:06.408 INFO:teuthology.orchestra.run.mira064.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-10-01T02:22:06.408 INFO:teuthology.orchestra.run.mira064.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-10-01T02:22:06.408 INFO:teuthology.orchestra.run.mira064.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-10-01T02:22:06.408 INFO:teuthology.orchestra.run.mira064.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-10-01T02:22:06.408 INFO:teuthology.orchestra.run.mira064.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-10-01T02:22:06.409 INFO:teuthology.orchestra.run.mira064.stdout: python2-kubernetes noarch 8.0.1-1.el7 epel 1.0 M 2019-10-01T02:22:06.409 INFO:teuthology.orchestra.run.mira064.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-10-01T02:22:06.409 INFO:teuthology.orchestra.run.mira064.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-10-01T02:22:06.409 INFO:teuthology.orchestra.run.mira064.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-10-01T02:22:06.410 INFO:teuthology.orchestra.run.mira064.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-10-01T02:22:06.410 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:06.410 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:06.410 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:06.411 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+10 Dependent packages) 2019-10-01T02:22:06.411 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:06.411 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 1.6 M 2019-10-01T02:22:06.411 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 22 M 2019-10-01T02:22:06.412 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:06.939 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:06.940 INFO:teuthology.orchestra.run.mira063.stdout:Total 2.6 MB/s | 1.6 MB 00:00 2019-10-01T02:22:06.940 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:06.941 INFO:teuthology.orchestra.run.mira064.stdout:Total 2.9 MB/s | 1.6 MB 00:00 2019-10-01T02:22:06.963 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:06.964 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:06.986 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:06.987 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:07.096 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:07.097 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:07.097 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:07.097 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:07.415 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 1/11 2019-10-01T02:22:07.415 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 1/11 2019-10-01T02:22:07.448 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 2/11 2019-10-01T02:22:07.449 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 2/11 2019-10-01T02:22:07.489 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 3/11 2019-10-01T02:22:07.489 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 3/11 2019-10-01T02:22:07.503 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 4/11 2019-10-01T02:22:07.504 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 4/11 2019-10-01T02:22:07.524 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-10-01T02:22:07.525 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-10-01T02:22:07.572 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-10-01T02:22:07.573 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-10-01T02:22:07.614 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-10-01T02:22:07.616 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-10-01T02:22:07.674 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-10-01T02:22:07.674 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-10-01T02:22:08.943 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-10-01T02:22:08.954 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-10-01T02:22:08.970 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python2-kubernetes-8.0.1-1.el7.noarch 10/11 2019-10-01T02:22:08.981 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python2-kubernetes-8.0.1-1.el7.noarch 10/11 2019-10-01T02:22:09.013 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 11/11 2019-10-01T02:22:09.016 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-10-01T02:22:09.018 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 11/11 2019-10-01T02:22:09.022 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2019-10-01T02:22:09.024 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-10-01T02:22:09.026 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 3/11 2019-10-01T02:22:09.030 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2019-10-01T02:22:09.030 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2019-10-01T02:22:09.035 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-5673.g2c0b216.el7.noarch 3/11 2019-10-01T02:22:09.035 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2019-10-01T02:22:09.038 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2019-10-01T02:22:09.040 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-10-01T02:22:09.043 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2019-10-01T02:22:09.044 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2019-10-01T02:22:09.047 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-10-01T02:22:09.049 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2019-10-01T02:22:09.052 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2019-10-01T02:22:09.054 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 9/11 2019-10-01T02:22:09.056 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2019-10-01T02:22:09.058 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2019-10-01T02:22:09.061 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 9/11 2019-10-01T02:22:09.066 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2019-10-01T02:22:09.252 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python2-kubernetes-8.0.1-1.el7.noarch 11/11 2019-10-01T02:22:09.252 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:09.252 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-rook.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-10-01T02:22:09.253 INFO:teuthology.orchestra.run.mira063.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-kubernetes.noarch 0:8.0.1-1.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-10-01T02:22:09.254 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:09.256 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:09.257 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python2-kubernetes-8.0.1-1.el7.noarch 11/11 2019-10-01T02:22:09.257 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-rook.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-10-01T02:22:09.258 INFO:teuthology.orchestra.run.mira064.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-kubernetes.noarch 0:8.0.1-1.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-10-01T02:22:09.259 INFO:teuthology.orchestra.run.mira064.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-10-01T02:22:09.260 INFO:teuthology.orchestra.run.mira064.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-10-01T02:22:09.260 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:09.260 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:09.359 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:09.360 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-mgr-ssh 2019-10-01T02:22:09.365 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:09.365 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-mgr-ssh 2019-10-01T02:22:09.648 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:09.652 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:09.724 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:09.726 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:09.727 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:09.727 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:09.727 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:09.728 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:09.729 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:09.730 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:09.730 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:09.731 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:11.171 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:11.172 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:11.787 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:11.788 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:11.788 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:11.790 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:11.790 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:11.791 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:11.791 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:22:11.793 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 2019-10-01T02:22:11.848 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:11.848 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-10-01T02:22:11.849 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-10-01T02:22:11.850 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:11.850 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-10-01T02:22:11.851 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-10-01T02:22:11.851 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:11.852 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-10-01T02:22:11.853 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:11.853 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-10-01T02:22:12.608 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:12.612 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:13.043 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:13.043 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-ssh noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 21 k 2019-10-01T02:22:13.045 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+2 Dependent packages) 2019-10-01T02:22:13.046 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:13.047 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:13.047 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:13.048 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 304 k 2019-10-01T02:22:13.048 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 1.0 M 2019-10-01T02:22:13.048 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:13.049 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-ssh noarch 2:15.0.0-5673.g2c0b216.el7 ceph-noarch 21 k 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+2 Dependent packages) 2019-10-01T02:22:13.050 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:13.051 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 304 k 2019-10-01T02:22:13.051 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 1.0 M 2019-10-01T02:22:13.051 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:13.406 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:13.407 INFO:teuthology.orchestra.run.mira064.stdout:Total 851 kB/s | 304 kB 00:00 2019-10-01T02:22:13.412 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:13.423 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:13.423 INFO:teuthology.orchestra.run.mira063.stdout:Total 807 kB/s | 304 kB 00:00 2019-10-01T02:22:13.424 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:13.429 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:13.440 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:13.549 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:13.549 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:13.552 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:13.552 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:13.809 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-10-01T02:22:13.822 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-10-01T02:22:13.850 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 3/3 2019-10-01T02:22:13.855 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 1/3 2019-10-01T02:22:13.861 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 2/3 2019-10-01T02:22:13.933 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-10-01T02:22:13.948 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-10-01T02:22:13.978 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 3/3 2019-10-01T02:22:13.985 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-5673.g2c0b216.el7.noarch 1/3 2019-10-01T02:22:13.991 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 2/3 2019-10-01T02:22:14.055 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 3/3 2019-10-01T02:22:14.055 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:14.055 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:14.055 INFO:teuthology.orchestra.run.mira063.stdout: ceph-mgr-ssh.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:14.055 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:14.056 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:22:14.056 INFO:teuthology.orchestra.run.mira063.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-10-01T02:22:14.056 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:14.056 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:14.166 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:14.166 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install ceph-fuse 2019-10-01T02:22:14.191 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 3/3 2019-10-01T02:22:14.191 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:14.191 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout: ceph-mgr-ssh.noarch 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:14.192 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:14.303 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:14.303 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install ceph-fuse 2019-10-01T02:22:14.428 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:14.504 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:14.506 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:14.507 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:14.507 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:14.508 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:14.581 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:14.657 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:14.659 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:14.659 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:14.660 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:14.660 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:15.952 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:16.096 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:16.576 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:16.577 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:16.577 INFO:teuthology.orchestra.run.mira063.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:16.597 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:16.659 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:16.661 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:16.661 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-10-01T02:22:16.662 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.668 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.668 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.670 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.671 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:16.671 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-10-01T02:22:16.682 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-10-01T02:22:16.682 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-10-01T02:22:16.715 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:16.715 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:16.715 INFO:teuthology.orchestra.run.mira064.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:16.736 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python36 for package: 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:16.798 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:16.800 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:16.800 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-10-01T02:22:16.802 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.805 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.807 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.809 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-10-01T02:22:16.810 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:16.811 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-10-01T02:22:16.821 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-10-01T02:22:16.822 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-10-01T02:22:17.427 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:17.567 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:17.858 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:17.858 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:17.860 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: ceph-fuse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 546 k 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-10-01T02:22:17.861 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:17.862 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:17.862 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:17.862 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:22:17.862 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:17.863 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 9.9 M 2019-10-01T02:22:17.863 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 49 M 2019-10-01T02:22:17.863 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:18.006 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:18.006 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:18.006 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:18.006 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout: ceph-fuse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 546 k 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:22:18.007 INFO:teuthology.orchestra.run.mira064.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:18.008 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:18.009 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+4 Dependent packages) 2019-10-01T02:22:18.009 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:18.009 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 9.9 M 2019-10-01T02:22:18.009 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 49 M 2019-10-01T02:22:18.010 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:18.374 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:18.375 INFO:teuthology.orchestra.run.mira063.stdout:Total 19 MB/s | 9.9 MB 00:00 2019-10-01T02:22:18.395 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:18.428 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:18.476 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:18.476 INFO:teuthology.orchestra.run.mira064.stdout:Total 21 MB/s | 9.9 MB 00:00 2019-10-01T02:22:18.501 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:18.539 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:18.608 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:18.609 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:18.736 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:18.737 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:21.040 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-10-01T02:22:21.174 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-10-01T02:22:21.279 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-10-01T02:22:21.413 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-10-01T02:22:22.362 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-10-01T02:22:22.380 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-10-01T02:22:22.460 INFO:teuthology.orchestra.run.mira064.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-10-01T02:22:22.475 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 5/5 2019-10-01T02:22:22.481 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/5 2019-10-01T02:22:22.496 INFO:teuthology.orchestra.run.mira063.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-10-01T02:22:22.497 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-10-01T02:22:22.502 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-10-01T02:22:22.507 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-10-01T02:22:22.512 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 5/5 2019-10-01T02:22:22.517 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:ceph-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/5 2019-10-01T02:22:22.535 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-10-01T02:22:22.541 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-10-01T02:22:22.546 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-10-01T02:22:22.702 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-10-01T02:22:22.702 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout: ceph-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-10-01T02:22:22.703 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:22.704 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:22.739 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-10-01T02:22:22.739 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:22.739 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:22.739 INFO:teuthology.orchestra.run.mira063.stdout: ceph-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:22.739 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:22.740 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:22:22.740 INFO:teuthology.orchestra.run.mira063.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-10-01T02:22:22.740 INFO:teuthology.orchestra.run.mira063.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-10-01T02:22:22.740 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:22.740 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:22.814 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:22.814 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install libcephfs2 2019-10-01T02:22:22.845 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:22.846 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install libcephfs2 2019-10-01T02:22:23.154 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:23.158 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:23.230 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:23.232 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:23.232 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:23.233 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:23.233 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:23.233 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:23.235 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:23.236 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:23.236 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:23.237 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:24.662 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:24.672 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:25.273 INFO:teuthology.orchestra.run.mira063.stdout:Package 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:25.273 INFO:teuthology.orchestra.run.mira063.stdout:Nothing to do 2019-10-01T02:22:25.281 INFO:teuthology.orchestra.run.mira064.stdout:Package 2:libcephfs2-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:25.281 INFO:teuthology.orchestra.run.mira064.stdout:Nothing to do 2019-10-01T02:22:25.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:25.368 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install libcephfs-devel 2019-10-01T02:22:25.375 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:25.375 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install libcephfs-devel 2019-10-01T02:22:25.643 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:25.719 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:25.720 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:25.721 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:25.722 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:25.722 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:25.723 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:25.796 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:25.798 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:25.799 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:25.800 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:25.800 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:27.164 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:27.242 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:27.781 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:27.782 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:27.782 INFO:teuthology.orchestra.run.mira064.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:27.784 INFO:teuthology.orchestra.run.mira064.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-5673.g2c0b216.el7 for package: 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:27.840 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:27.841 INFO:teuthology.orchestra.run.mira064.stdout:---> Package librados-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:27.858 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:27.859 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:27.859 INFO:teuthology.orchestra.run.mira063.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:27.863 INFO:teuthology.orchestra.run.mira063.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-5673.g2c0b216.el7 for package: 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 2019-10-01T02:22:27.969 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:27.969 INFO:teuthology.orchestra.run.mira063.stdout:---> Package librados-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:28.614 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:28.765 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:29.050 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.051 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:29.051 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout: libcephfs-devel x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 17 k 2019-10-01T02:22:29.052 INFO:teuthology.orchestra.run.mira064.stdout:Installing for dependencies: 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout: librados-devel x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 85 k 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package (+1 Dependent package) 2019-10-01T02:22:29.053 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.054 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 101 k 2019-10-01T02:22:29.054 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 372 k 2019-10-01T02:22:29.054 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:29.197 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:29.197 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:29.198 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:29.198 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:29.198 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout: libcephfs-devel x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 17 k 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout:Installing for dependencies: 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout: librados-devel x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 85 k 2019-10-01T02:22:29.199 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:29.200 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:29.200 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:29.200 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package (+1 Dependent package) 2019-10-01T02:22:29.200 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:29.201 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 101 k 2019-10-01T02:22:29.201 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 372 k 2019-10-01T02:22:29.201 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:29.524 INFO:teuthology.orchestra.run.mira064.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:29.524 INFO:teuthology.orchestra.run.mira064.stdout:Total 215 kB/s | 101 kB 00:00 2019-10-01T02:22:29.527 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:29.548 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:29.551 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:29.552 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:29.634 INFO:teuthology.orchestra.run.mira063.stdout:-------------------------------------------------------------------------------- 2019-10-01T02:22:29.634 INFO:teuthology.orchestra.run.mira063.stdout:Total 233 kB/s | 101 kB 00:00 2019-10-01T02:22:29.638 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:29.671 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:29.675 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:29.676 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:29.681 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:librados-devel-15.0.0-5673.g2c0b216.el7.x86_64 1/2 2019-10-01T02:22:29.695 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 2/2 2019-10-01T02:22:29.702 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 1/2 2019-10-01T02:22:29.793 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:librados-devel-15.0.0-5673.g2c0b216.el7.x86_64 1/2 2019-10-01T02:22:29.805 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 2/2 2019-10-01T02:22:29.812 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:libcephfs-devel-15.0.0-5673.g2c0b216.el7.x86_64 1/2 2019-10-01T02:22:29.911 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:librados-devel-15.0.0-5673.g2c0b216.el7.x86_64 2/2 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout: libcephfs-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout:Dependency Installed: 2019-10-01T02:22:29.912 INFO:teuthology.orchestra.run.mira064.stdout: librados-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:29.913 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:29.913 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:30.010 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:librados-devel-15.0.0-5673.g2c0b216.el7.x86_64 2/2 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout: libcephfs-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout:Dependency Installed: 2019-10-01T02:22:30.011 INFO:teuthology.orchestra.run.mira063.stdout: librados-devel.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:30.012 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:30.012 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:30.022 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:30.023 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install librados2 2019-10-01T02:22:30.117 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:30.118 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install librados2 2019-10-01T02:22:30.321 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:30.397 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:30.405 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:30.405 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:30.405 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:30.405 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:30.430 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:30.506 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:30.508 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:30.509 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:30.509 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:30.510 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:31.845 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:31.946 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:32.458 INFO:teuthology.orchestra.run.mira064.stdout:Package 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:32.459 INFO:teuthology.orchestra.run.mira064.stdout:Nothing to do 2019-10-01T02:22:32.554 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:32.554 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install librbd1 2019-10-01T02:22:32.560 INFO:teuthology.orchestra.run.mira063.stdout:Package 2:librados2-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:32.560 INFO:teuthology.orchestra.run.mira063.stdout:Nothing to do 2019-10-01T02:22:32.656 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:32.656 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install librbd1 2019-10-01T02:22:32.902 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:32.912 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:32.979 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:32.980 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:32.981 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:32.982 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:32.982 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:32.988 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:32.990 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:32.991 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:32.991 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:32.992 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:34.432 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:34.465 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:35.044 INFO:teuthology.orchestra.run.mira064.stdout:Package 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:35.045 INFO:teuthology.orchestra.run.mira064.stdout:Nothing to do 2019-10-01T02:22:35.081 INFO:teuthology.orchestra.run.mira063.stdout:Package 2:librbd1-15.0.0-5673.g2c0b216.el7.x86_64 already installed and latest version 2019-10-01T02:22:35.081 INFO:teuthology.orchestra.run.mira063.stdout:Nothing to do 2019-10-01T02:22:35.139 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:35.139 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install python-ceph 2019-10-01T02:22:35.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:35.176 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install python-ceph 2019-10-01T02:22:35.442 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:35.449 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:35.518 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:35.520 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:35.520 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:35.521 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:35.521 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:35.522 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:35.524 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:35.525 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:35.525 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:35.526 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:36.953 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:36.969 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:37.566 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:37.567 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:37.567 INFO:teuthology.orchestra.run.mira063.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:37.583 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:37.583 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:37.584 INFO:teuthology.orchestra.run.mira064.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:38.347 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:38.350 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:38.781 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:38.781 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout: Size 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:38.782 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-compat x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 2.2 k 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:38.783 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 2.2 k 2019-10-01T02:22:38.784 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 0 2019-10-01T02:22:38.784 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:38.785 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:38.785 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:38.786 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:38.786 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:38.786 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout: Size 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-compat x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 2.2 k 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:38.787 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:38.788 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package 2019-10-01T02:22:38.788 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:38.788 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 2.2 k 2019-10-01T02:22:38.789 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 0 2019-10-01T02:22:38.789 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:39.157 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:39.158 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:39.159 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:39.160 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:39.167 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:39.168 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:39.173 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:39.173 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:39.263 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:python-ceph-compat-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:39.272 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:python-ceph-compat-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:39.489 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:python-ceph-compat-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:39.489 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:39.490 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:39.490 INFO:teuthology.orchestra.run.mira064.stdout: python-ceph-compat.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:39.490 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:39.490 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:39.494 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:python-ceph-compat-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:39.494 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:39.494 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:39.495 INFO:teuthology.orchestra.run.mira063.stdout: python-ceph-compat.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:39.495 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:39.495 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:39.597 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:39.597 INFO:teuthology.orchestra.run.mira064:> sudo yum -y install rbd-fuse 2019-10-01T02:22:39.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:39.600 INFO:teuthology.orchestra.run.mira063:> sudo yum -y install rbd-fuse 2019-10-01T02:22:39.965 INFO:teuthology.orchestra.run.mira063.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:39.969 INFO:teuthology.orchestra.run.mira064.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-10-01T02:22:40.042 INFO:teuthology.orchestra.run.mira063.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:40.044 INFO:teuthology.orchestra.run.mira063.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:40.044 INFO:teuthology.orchestra.run.mira063.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:40.045 INFO:teuthology.orchestra.run.mira063.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:40.046 INFO:teuthology.orchestra.run.mira064.stdout:Loading mirror speeds from cached hostfile 2019-10-01T02:22:40.046 INFO:teuthology.orchestra.run.mira063.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:40.055 INFO:teuthology.orchestra.run.mira064.stdout: * base: mirror.vtti.vt.edu 2019-10-01T02:22:40.055 INFO:teuthology.orchestra.run.mira064.stdout: * epel: ftp.linux.ncsu.edu 2019-10-01T02:22:40.055 INFO:teuthology.orchestra.run.mira064.stdout: * extras: mirror.fileplanet.com 2019-10-01T02:22:40.055 INFO:teuthology.orchestra.run.mira064.stdout: * updates: mirror.jaleco.com 2019-10-01T02:22:41.486 INFO:teuthology.orchestra.run.mira063.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:41.486 INFO:teuthology.orchestra.run.mira064.stdout:7 packages excluded due to repository priority protections 2019-10-01T02:22:42.100 INFO:teuthology.orchestra.run.mira063.stdout:Resolving Dependencies 2019-10-01T02:22:42.101 INFO:teuthology.orchestra.run.mira063.stdout:--> Running transaction check 2019-10-01T02:22:42.101 INFO:teuthology.orchestra.run.mira063.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:42.102 INFO:teuthology.orchestra.run.mira064.stdout:Resolving Dependencies 2019-10-01T02:22:42.102 INFO:teuthology.orchestra.run.mira064.stdout:--> Running transaction check 2019-10-01T02:22:42.102 INFO:teuthology.orchestra.run.mira064.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 will be installed 2019-10-01T02:22:42.874 INFO:teuthology.orchestra.run.mira063.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:42.879 INFO:teuthology.orchestra.run.mira064.stdout:--> Finished Dependency Resolution 2019-10-01T02:22:43.312 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:43.313 INFO:teuthology.orchestra.run.mira063.stdout:Dependencies Resolved 2019-10-01T02:22:43.314 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:43.314 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:43.314 INFO:teuthology.orchestra.run.mira063.stdout: Package Arch Version Repository Size 2019-10-01T02:22:43.314 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:43.314 INFO:teuthology.orchestra.run.mira063.stdout:Installing: 2019-10-01T02:22:43.315 INFO:teuthology.orchestra.run.mira063.stdout: rbd-fuse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 67 k 2019-10-01T02:22:43.315 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:43.315 INFO:teuthology.orchestra.run.mira063.stdout:Transaction Summary 2019-10-01T02:22:43.315 INFO:teuthology.orchestra.run.mira063.stdout:================================================================================ 2019-10-01T02:22:43.315 INFO:teuthology.orchestra.run.mira063.stdout:Install 1 Package 2019-10-01T02:22:43.316 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:43.316 INFO:teuthology.orchestra.run.mira063.stdout:Total download size: 67 k 2019-10-01T02:22:43.316 INFO:teuthology.orchestra.run.mira063.stdout:Installed size: 172 k 2019-10-01T02:22:43.316 INFO:teuthology.orchestra.run.mira063.stdout:Downloading packages: 2019-10-01T02:22:43.323 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:43.323 INFO:teuthology.orchestra.run.mira064.stdout:Dependencies Resolved 2019-10-01T02:22:43.323 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:43.323 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout: Package Arch Version Repository Size 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout:Installing: 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout: rbd-fuse x86_64 2:15.0.0-5673.g2c0b216.el7 ceph 67 k 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout:Transaction Summary 2019-10-01T02:22:43.324 INFO:teuthology.orchestra.run.mira064.stdout:================================================================================ 2019-10-01T02:22:43.325 INFO:teuthology.orchestra.run.mira064.stdout:Install 1 Package 2019-10-01T02:22:43.325 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:43.325 INFO:teuthology.orchestra.run.mira064.stdout:Total download size: 67 k 2019-10-01T02:22:43.326 INFO:teuthology.orchestra.run.mira064.stdout:Installed size: 172 k 2019-10-01T02:22:43.326 INFO:teuthology.orchestra.run.mira064.stdout:Downloading packages: 2019-10-01T02:22:43.726 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction check 2019-10-01T02:22:43.740 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction test 2019-10-01T02:22:43.742 INFO:teuthology.orchestra.run.mira063.stdout:Transaction test succeeded 2019-10-01T02:22:43.742 INFO:teuthology.orchestra.run.mira063.stdout:Running transaction 2019-10-01T02:22:43.743 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction check 2019-10-01T02:22:43.758 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction test 2019-10-01T02:22:43.759 INFO:teuthology.orchestra.run.mira064.stdout:Transaction test succeeded 2019-10-01T02:22:43.759 INFO:teuthology.orchestra.run.mira064.stdout:Running transaction 2019-10-01T02:22:43.861 INFO:teuthology.orchestra.run.mira063.stdout: Installing : 2:rbd-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:43.873 INFO:teuthology.orchestra.run.mira064.stdout: Installing : 2:rbd-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:44.076 INFO:teuthology.orchestra.run.mira063.stdout: Verifying : 2:rbd-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:44.076 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:44.077 INFO:teuthology.orchestra.run.mira063.stdout:Installed: 2019-10-01T02:22:44.077 INFO:teuthology.orchestra.run.mira063.stdout: rbd-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:44.077 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:22:44.077 INFO:teuthology.orchestra.run.mira063.stdout:Complete! 2019-10-01T02:22:44.132 INFO:teuthology.orchestra.run.mira064.stdout: Verifying : 2:rbd-fuse-15.0.0-5673.g2c0b216.el7.x86_64 1/1 2019-10-01T02:22:44.132 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:44.132 INFO:teuthology.orchestra.run.mira064.stdout:Installed: 2019-10-01T02:22:44.132 INFO:teuthology.orchestra.run.mira064.stdout: rbd-fuse.x86_64 2:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:44.132 INFO:teuthology.orchestra.run.mira064.stdout: 2019-10-01T02:22:44.133 INFO:teuthology.orchestra.run.mira064.stdout:Complete! 2019-10-01T02:22:44.185 DEBUG:teuthology.parallel:result is None 2019-10-01T02:22:44.239 DEBUG:teuthology.parallel:result is None 2019-10-01T02:22:44.240 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:44.240 INFO:teuthology.orchestra.run.mira063:> sudo lsb_release -is 2019-10-01T02:22:44.388 INFO:teuthology.orchestra.run.mira063.stdout:CentOS 2019-10-01T02:22:44.391 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:22:44.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=2c0b2169f6303e11ab6bed736272e5e2d23ab027 2019-10-01T02:22:44.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:44.559 INFO:teuthology.orchestra.run.mira063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-10-01T02:22:44.729 INFO:teuthology.orchestra.run.mira063.stdout:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:44.730 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:44.730 INFO:teuthology.task.install:The correct ceph version 15.0.0-5673.g2c0b216 is installed. 2019-10-01T02:22:44.731 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:44.731 INFO:teuthology.orchestra.run.mira064:> sudo lsb_release -is 2019-10-01T02:22:44.890 INFO:teuthology.orchestra.run.mira064.stdout:CentOS 2019-10-01T02:22:44.893 DEBUG:teuthology.misc:System to be installed: CentOS 2019-10-01T02:22:44.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=2c0b2169f6303e11ab6bed736272e5e2d23ab027 2019-10-01T02:22:45.072 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:45.072 INFO:teuthology.orchestra.run.mira064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-10-01T02:22:45.236 INFO:teuthology.orchestra.run.mira064.stdout:15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:45.237 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-5673.g2c0b216.el7 2019-10-01T02:22:45.237 INFO:teuthology.task.install:The correct ceph version 15.0.0-5673.g2c0b216 is installed. 2019-10-01T02:22:45.237 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-10-01T02:22:45.238 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:45.238 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-01T02:22:45.398 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:45.399 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-01T02:22:45.542 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-10-01T02:22:45.542 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:45.542 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-01T02:22:45.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:45.702 INFO:teuthology.orchestra.run.mira063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-01T02:22:45.846 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:45.846 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-01T02:22:45.995 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:45.996 INFO:teuthology.orchestra.run.mira064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-01T02:22:46.155 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-10-01T02:22:46.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:46.156 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-01T02:22:46.304 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:46.304 INFO:teuthology.orchestra.run.mira063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-01T02:22:46.453 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:46.453 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-01T02:22:46.617 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:46.617 INFO:teuthology.orchestra.run.mira064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-01T02:22:46.781 INFO:teuthology.run_tasks:Running task ceph... 2019-10-01T02:22:47.002 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-10-01T02:22:47.002 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.002 INFO:teuthology.orchestra.run.mira063:> sudo chmod 777 /var/log/ceph 2019-10-01T02:22:47.009 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:47.009 INFO:teuthology.orchestra.run.mira064:> sudo chmod 777 /var/log/ceph 2019-10-01T02:22:47.164 INFO:tasks.ceph:Disabling ceph logrotate... 2019-10-01T02:22:47.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.165 INFO:teuthology.orchestra.run.mira063:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-01T02:22:47.171 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:47.171 INFO:teuthology.orchestra.run.mira064:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-01T02:22:47.327 INFO:tasks.ceph:Creating extra log directories... 2019-10-01T02:22:47.328 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.328 INFO:teuthology.orchestra.run.mira063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-01T02:22:47.337 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:47.337 INFO:teuthology.orchestra.run.mira064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-01T02:22:47.498 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-10-01T02:22:47.498 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_DOWN\\)'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'ms inject socket failures': 5000, 'ms type': 'async'}, 'mon': {'paxos min': 10, 'debug mon': 20, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos trim min': 10, 'paxos service trim min': 5}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'bluestore allocator': 'stupid', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'bdev async discard': True, 'osd op queue': 'debug_random', 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}}} 2019-10-01T02:22:47.498 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'mira')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_many': True, 'thrash_delay': 1}}, {'rados': {'op_weights': {'snap_remove': 50, 'write': 100, 'snap_create': 50, 'rollback': 50, 'read': 100, 'delete': 50, 'copy_from': 50}, 'clients': ['client.0'], 'objects': 50, 'ops': 4000}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.mira.11875', 'owner': 'scheduled_kchai@teuthology', 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'targets': {'mira063.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC0syhJayVLaGcvkxkrH7lkHRQ+9cOZOrjrmRflouJyaEdL6Qo0rCPCL+2XUeLe7iZbSquaSvxF/shLm7hQWBC2wNMPB+BY//Xe0v9kcN7W7IJ7TEkOuQBHx6mWftT5ncTcmsyZtixRos+ejLoQCqJdvNVIAaqHExwvgvFPFRJGloz7ceptcY90+IkonAxuSvtO3vmeEvTb1q5bDZpd4mveI1+XlTLyFhMuLziYImO5UBvStnBGbgRV6BaNQcNvRF59VF25S/nHBQtm3FUGcF5mqdARwJq95F6v7ZP0FVdOyG5JZ0j/XaIXeQysaG3AFNNUCggjNHZKhMxX1+L4KFf', 'mira064.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE+Ux0+98wzMAe6H0MZQA31ni2vyP0dNZ8EZAFJBqpq0/AITvn7JrRa3WSI6FItIYUdS3tzogVRnnQCjq4/I4CWwxw9pFpYSc0Ikx3r8TVo5qqqrSGkcSRo6DCFZe/GVQZaAAhclynmA/1oPxr62FD7ohW9oWlHOZE/vtcqxKiTjxXTrsxE31qRh8VA3V8x/5/ZijfORMjqQKWKhkJ1oPOOGqi81G7cTatOnA4Ri1PCB8UO34p5MX9ESSUFqaw+BPql5olSoenD3aIEeCCLofuNQ85Gh9nwQSKG+Q08Fb9HSzD3QRBl9JE0j65ZUr8Z6Pf37qv6wg9NNGXTm2Coq69'}, 'job_id': '4347574', 'tube': 'mira', 'suite_branch': 'wip-kefu-testing-2019-09-30-1336', 'priority': 70, 'os_version': '7.6', 'first_in_suite': False, 'branch': 'wip-kefu-testing-2019-09-30-1336', 'suite': 'rados', 'email': 'tchaikov@gmail.com', 'archive_path': '/home/teuthworker/archive/kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira/4347574', 'description': 'rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'mira', 'teuthology_branch': 'master', 'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027', 'name': 'kchai-2019-09-30_08:32:30-rados-wip-kefu-testing-2019-09-30-1336-distro-basic-mira', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027', 'os_type': 'centos', 'overrides': {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_DOWN\\)'], 'conf': {'global': {'ms inject socket failures': 5000, 'ms type': 'async'}, 'mon': {'paxos min': 10, 'debug mon': 20, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos trim min': 10, 'paxos service trim min': 5}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'bluestore allocator': 'stupid', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'bdev async discard': True, 'osd op queue': 'debug_random', 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}}, 'fs': 'xfs', 'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027'}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95, 'bdev async discard': True}}}, 'install': {'ceph': {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027'}}, 'admin_socket': {'branch': 'wip-kefu-testing-2019-09-30-1336'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': '2c0b2169f6303e11ab6bed736272e5e2d23ab027', 'branch': 'wip-kefu-testing-2019-09-30-1336'}}} 2019-10-01T02:22:47.499 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.499 INFO:teuthology.orchestra.run.mira063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-01T02:22:47.505 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:47.505 INFO:teuthology.orchestra.run.mira064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-01T02:22:47.617 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.618 INFO:teuthology.orchestra.run.mira063:> sudo install -d -m0777 -- /var/run/ceph 2019-10-01T02:22:47.623 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:47.623 INFO:teuthology.orchestra.run.mira064:> sudo install -d -m0777 -- /var/run/ceph 2019-10-01T02:22:47.898 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:47.898 INFO:teuthology.orchestra.run.mira063:> ls /dev/[sv]d? 2019-10-01T02:22:48.010 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sda 2019-10-01T02:22:48.010 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdb 2019-10-01T02:22:48.010 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdc 2019-10-01T02:22:48.011 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdd 2019-10-01T02:22:48.011 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sde 2019-10-01T02:22:48.011 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdf 2019-10-01T02:22:48.011 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdg 2019-10-01T02:22:48.011 INFO:teuthology.orchestra.run.mira063.stdout:/dev/sdh 2019-10-01T02:22:48.012 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh'] 2019-10-01T02:22:48.012 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.013 INFO:teuthology.orchestra.run.mira063:> stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda 2019-10-01T02:22:48.125 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sda’ 2019-10-01T02:22:48.125 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.126 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12685 Links: 1 Device type: 8,0 2019-10-01T02:22:48.126 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.126 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.126 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:10:20.123727175 +0000 2019-10-01T02:22:48.127 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:10:02.007484316 +0000 2019-10-01T02:22:48.127 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:10:02.007484316 +0000 2019-10-01T02:22:48.127 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:48.173 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:48.173 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:48.173 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000274888 s, 1.9 MB/s 2019-10-01T02:22:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-01T02:22:48.181 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2019-10-01T02:22:48.181 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.181 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb 2019-10-01T02:22:48.280 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdb’ 2019-10-01T02:22:48.280 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12043 Links: 1 Device type: 8,10 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:34.247727944 +0000 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:34.247727944 +0000 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:34.247727944 +0000 2019-10-01T02:22:48.281 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:48.327 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:48.327 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:48.327 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000750323 s, 682 kB/s 2019-10-01T02:22:48.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.338 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc 2019-10-01T02:22:48.450 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdc’ 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12046 Links: 1 Device type: 8,20 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:35.293751161 +0000 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:35.293751161 +0000 2019-10-01T02:22:48.451 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:35.293751161 +0000 2019-10-01T02:22:48.452 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:48.499 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:48.500 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:48.500 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000756479 s, 677 kB/s 2019-10-01T02:22:48.512 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.512 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd 2019-10-01T02:22:48.630 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdd’ 2019-10-01T02:22:48.631 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.631 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12051 Links: 1 Device type: 8,30 2019-10-01T02:22:48.631 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.631 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.632 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:30.134636637 +0000 2019-10-01T02:22:48.632 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:30.134636637 +0000 2019-10-01T02:22:48.632 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:30.134636637 +0000 2019-10-01T02:22:48.632 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:48.677 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:48.677 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:48.678 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000731471 s, 700 kB/s 2019-10-01T02:22:48.685 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.685 INFO:teuthology.orchestra.run.mira063:> stat /dev/sde && sudo dd if=/dev/sde of=/dev/null count=1 && ! mount | grep -q /dev/sde 2019-10-01T02:22:48.799 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sde’ 2019-10-01T02:22:48.799 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.799 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12055 Links: 1 Device type: 8,40 2019-10-01T02:22:48.799 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.799 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.800 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:31.612669447 +0000 2019-10-01T02:22:48.800 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:31.612669447 +0000 2019-10-01T02:22:48.800 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:31.612669447 +0000 2019-10-01T02:22:48.800 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:48.847 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:48.847 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:48.847 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000759225 s, 674 kB/s 2019-10-01T02:22:48.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:48.854 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdf && sudo dd if=/dev/sdf of=/dev/null count=1 && ! mount | grep -q /dev/sdf 2019-10-01T02:22:48.965 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdf’ 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12052 Links: 1 Device type: 8,50 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:42.450910047 +0000 2019-10-01T02:22:48.966 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:34.099724656 +0000 2019-10-01T02:22:48.967 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:34.099724656 +0000 2019-10-01T02:22:48.967 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:49.021 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:49.022 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:49.022 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.0113303 s, 45.2 kB/s 2019-10-01T02:22:49.029 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:49.029 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdg && sudo dd if=/dev/sdg of=/dev/null count=1 && ! mount | grep -q /dev/sdg 2019-10-01T02:22:49.139 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdg’ 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12053 Links: 1 Device type: 8,60 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:32.568690669 +0000 2019-10-01T02:22:49.140 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:32.568690669 +0000 2019-10-01T02:22:49.141 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:32.568690669 +0000 2019-10-01T02:22:49.141 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:49.188 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:49.188 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:49.188 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.000697221 s, 734 kB/s 2019-10-01T02:22:49.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:49.195 INFO:teuthology.orchestra.run.mira063:> stat /dev/sdh && sudo dd if=/dev/sdh of=/dev/null count=1 && ! mount | grep -q /dev/sdh 2019-10-01T02:22:49.300 INFO:teuthology.orchestra.run.mira063.stdout: File: ‘/dev/sdh’ 2019-10-01T02:22:49.300 INFO:teuthology.orchestra.run.mira063.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Device: 5h/5d Inode: 12054 Links: 1 Device type: 8,70 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Access: 2019-10-01 02:18:47.815029113 +0000 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Modify: 2019-10-01 02:18:47.815029113 +0000 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout:Change: 2019-10-01 02:18:47.815029113 +0000 2019-10-01T02:22:49.301 INFO:teuthology.orchestra.run.mira063.stdout: Birth: - 2019-10-01T02:22:49.366 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records in 2019-10-01T02:22:49.366 INFO:teuthology.orchestra.run.mira063.stderr:1+0 records out 2019-10-01T02:22:49.366 INFO:teuthology.orchestra.run.mira063.stderr:512 bytes (512 B) copied, 0.0188979 s, 27.1 kB/s 2019-10-01T02:22:49.373 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-01T02:22:49.373 INFO:tasks.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh'] 2019-10-01T02:22:49.373 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:49.373 INFO:teuthology.orchestra.run.mira063:> ls -l '/dev/disk/by-id/wwn-*' 2019-10-01T02:22:49.474 INFO:teuthology.orchestra.run.mira063.stderr:ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2019-10-01T02:22:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-10-01T02:22:49.474 INFO:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2019-10-01T02:22:49.475 INFO:tasks.ceph:dev map: {'osd.1': '/dev/sdf', 'osd.0': '/dev/sdh', 'osd.2': '/dev/sdg'} 2019-10-01T02:22:49.599 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:49.600 INFO:teuthology.orchestra.run.mira064:> ls /dev/[sv]d? 2019-10-01T02:22:49.715 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sda 2019-10-01T02:22:49.715 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdb 2019-10-01T02:22:49.716 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdc 2019-10-01T02:22:49.716 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdd 2019-10-01T02:22:49.716 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sde 2019-10-01T02:22:49.716 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdf 2019-10-01T02:22:49.717 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdg 2019-10-01T02:22:49.717 INFO:teuthology.orchestra.run.mira064.stdout:/dev/sdh 2019-10-01T02:22:49.717 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh'] 2019-10-01T02:22:49.718 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:49.718 INFO:teuthology.orchestra.run.mira064:> stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda 2019-10-01T02:22:49.838 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sda’ 2019-10-01T02:22:49.838 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:49.838 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15059 Links: 1 Device type: 8,0 2019-10-01T02:22:49.838 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:49.838 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:49.839 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:10:25.898392266 +0000 2019-10-01T02:22:49.839 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:10:09.166750621 +0000 2019-10-01T02:22:49.839 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:10:09.166750621 +0000 2019-10-01T02:22:49.839 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:49.883 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:49.883 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:49.883 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.000314803 s, 1.6 MB/s 2019-10-01T02:22:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-01T02:22:49.891 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2019-10-01T02:22:49.891 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:49.891 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb 2019-10-01T02:22:49.994 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdb’ 2019-10-01T02:22:49.994 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:49.994 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15028 Links: 1 Device type: 8,10 2019-10-01T02:22:49.994 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:49.995 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:49.995 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:35.646281763 +0000 2019-10-01T02:22:49.995 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:35.646281763 +0000 2019-10-01T02:22:49.995 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:35.646281763 +0000 2019-10-01T02:22:49.996 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.055 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.055 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.055 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.0144974 s, 35.3 kB/s 2019-10-01T02:22:50.062 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.062 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc 2019-10-01T02:22:50.163 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdc’ 2019-10-01T02:22:50.163 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:50.164 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15033 Links: 1 Device type: 8,20 2019-10-01T02:22:50.164 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:50.164 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:50.164 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:36.366286967 +0000 2019-10-01T02:22:50.165 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:36.366286967 +0000 2019-10-01T02:22:50.165 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:36.366286967 +0000 2019-10-01T02:22:50.165 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.223 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.224 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.224 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.0140223 s, 36.5 kB/s 2019-10-01T02:22:50.231 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.231 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd 2019-10-01T02:22:50.339 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdd’ 2019-10-01T02:22:50.339 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15036 Links: 1 Device type: 8,30 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:32.060255845 +0000 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:32.060255845 +0000 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:32.060255845 +0000 2019-10-01T02:22:50.340 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.388 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.388 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.388 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.000839323 s, 610 kB/s 2019-10-01T02:22:50.401 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.402 INFO:teuthology.orchestra.run.mira064:> stat /dev/sde && sudo dd if=/dev/sde of=/dev/null count=1 && ! mount | grep -q /dev/sde 2019-10-01T02:22:50.514 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sde’ 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15046 Links: 1 Device type: 8,40 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:33.571266766 +0000 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:33.571266766 +0000 2019-10-01T02:22:50.515 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:33.571266766 +0000 2019-10-01T02:22:50.516 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.560 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.560 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.560 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.000849359 s, 603 kB/s 2019-10-01T02:22:50.567 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.567 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdf && sudo dd if=/dev/sdf of=/dev/null count=1 && ! mount | grep -q /dev/sdf 2019-10-01T02:22:50.681 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdf’ 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15045 Links: 1 Device type: 8,50 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:34.208271370 +0000 2019-10-01T02:22:50.682 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:34.208271370 +0000 2019-10-01T02:22:50.683 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:34.208271370 +0000 2019-10-01T02:22:50.683 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.729 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.729 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.729 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.000904406 s, 566 kB/s 2019-10-01T02:22:50.736 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.736 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdg && sudo dd if=/dev/sdg of=/dev/null count=1 && ! mount | grep -q /dev/sdg 2019-10-01T02:22:50.848 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdg’ 2019-10-01T02:22:50.849 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:50.849 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15043 Links: 1 Device type: 8,60 2019-10-01T02:22:50.849 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:50.849 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:50.850 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:34.905276407 +0000 2019-10-01T02:22:50.850 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:34.905276407 +0000 2019-10-01T02:22:50.851 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:34.905276407 +0000 2019-10-01T02:22:50.852 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:50.897 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:50.897 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:50.897 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.000913066 s, 561 kB/s 2019-10-01T02:22:50.905 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:50.905 INFO:teuthology.orchestra.run.mira064:> stat /dev/sdh && sudo dd if=/dev/sdh of=/dev/null count=1 && ! mount | grep -q /dev/sdh 2019-10-01T02:22:51.018 INFO:teuthology.orchestra.run.mira064.stdout: File: ‘/dev/sdh’ 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout:Device: 5h/5d Inode: 15044 Links: 1 Device type: 8,70 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout:Access: 2019-10-01 02:18:37.024291723 +0000 2019-10-01T02:22:51.019 INFO:teuthology.orchestra.run.mira064.stdout:Modify: 2019-10-01 02:18:37.024291723 +0000 2019-10-01T02:22:51.020 INFO:teuthology.orchestra.run.mira064.stdout:Change: 2019-10-01 02:18:37.024291723 +0000 2019-10-01T02:22:51.020 INFO:teuthology.orchestra.run.mira064.stdout: Birth: - 2019-10-01T02:22:51.077 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records in 2019-10-01T02:22:51.077 INFO:teuthology.orchestra.run.mira064.stderr:1+0 records out 2019-10-01T02:22:51.077 INFO:teuthology.orchestra.run.mira064.stderr:512 bytes (512 B) copied, 0.00991777 s, 51.6 kB/s 2019-10-01T02:22:51.084 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-01T02:22:51.084 INFO:tasks.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh'] 2019-10-01T02:22:51.085 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:51.085 INFO:teuthology.orchestra.run.mira064:> ls -l '/dev/disk/by-id/wwn-*' 2019-10-01T02:22:51.200 INFO:teuthology.orchestra.run.mira064.stderr:ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2019-10-01T02:22:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-10-01T02:22:51.200 INFO:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2019-10-01T02:22:51.201 INFO:tasks.ceph:dev map: {'osd.3': '/dev/sdh', 'osd.5': '/dev/sdg', 'osd.4': '/dev/sdf'} 2019-10-01T02:22:51.201 INFO:tasks.ceph:Generating config... 2019-10-01T02:22:51.202 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2019-10-01T02:22:51.202 INFO:tasks.ceph:[global] ms type = async 2019-10-01T02:22:51.202 INFO:tasks.ceph:[mon] paxos min = 10 2019-10-01T02:22:51.202 INFO:tasks.ceph:[mon] debug mon = 20 2019-10-01T02:22:51.202 INFO:tasks.ceph:[mon] debug paxos = 20 2019-10-01T02:22:51.202 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] debug ms = 1 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] paxos trim min = 10 2019-10-01T02:22:51.203 INFO:tasks.ceph:[mon] paxos service trim min = 5 2019-10-01T02:22:51.203 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-10-01T02:22:51.203 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] debug ms = 1 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] debug osd = 25 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-10-01T02:22:51.204 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] debug journal = 20 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2019-10-01T02:22:51.205 INFO:tasks.ceph:[osd] bdev async discard = True 2019-10-01T02:22:51.206 INFO:tasks.ceph:[osd] osd op queue = debug_random 2019-10-01T02:22:51.206 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-10-01T02:22:51.206 INFO:tasks.ceph:[osd] debug filestore = 20 2019-10-01T02:22:51.206 INFO:tasks.ceph:[osd] bdev enable discard = True 2019-10-01T02:22:51.206 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-10-01T02:22:51.206 INFO:tasks.ceph:Setting up mon.a... 2019-10-01T02:22:51.206 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:51.207 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-10-01T02:22:51.424 INFO:teuthology.orchestra.run.mira063.stdout:creating /etc/ceph/ceph.keyring 2019-10-01T02:22:51.430 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:51.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-10-01T02:22:51.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:51.643 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-01T02:22:51.800 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.6.124'), ('mon.c', '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]'), ('mon.b', '172.21.6.120')] 2019-10-01T02:22:51.801 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:51.801 INFO:teuthology.orchestra.run.mira063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.6.124 --addv c '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' --add b 172.21.6.120 --print /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:22:51.995 INFO:teuthology.orchestra.run.mira063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:monmaptool: generated fsid 3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:epoch 0 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:fsid 3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:last_changed 2019-10-01T02:22:51.990140+0000 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:created 2019-10-01T02:22:51.990140+0000 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:min_mon_release 0 (unknown) 2019-10-01T02:22:51.996 INFO:teuthology.orchestra.run.mira063.stdout:0: [v2:172.21.6.124:3300/0,v1:172.21.6.124:6789/0] mon.a 2019-10-01T02:22:51.997 INFO:teuthology.orchestra.run.mira063.stdout:1: [v2:172.21.6.120:3300/0,v1:172.21.6.120:6789/0] mon.b 2019-10-01T02:22:51.997 INFO:teuthology.orchestra.run.mira063.stdout:2: [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] mon.c 2019-10-01T02:22:51.997 INFO:teuthology.orchestra.run.mira063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-10-01T02:22:52.001 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525... 2019-10-01T02:22:52.002 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:52.002 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-01T02:22:52.014 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:52.014 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-01T02:22:52.350 INFO:tasks.ceph:Creating admin key on mon.a... 2019-10-01T02:22:52.350 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:52.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-10-01T02:22:52.603 INFO:tasks.ceph:Copying monmap to all nodes... 2019-10-01T02:22:52.704 DEBUG:teuthology.orchestra.remote:mira063:/etc/ceph/ceph.keyring is 206B 2019-10-01T02:22:52.902 DEBUG:teuthology.orchestra.remote:mira063:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-10-01T02:22:53.023 INFO:tasks.ceph:Sending monmap to node ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:22:53.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:53.023 INFO:teuthology.orchestra.run.mira063:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-01T02:22:53.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:53.235 INFO:teuthology.orchestra.run.mira063:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:22:53.348 INFO:tasks.ceph:Sending monmap to node ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:22:53.348 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:53.348 INFO:teuthology.orchestra.run.mira064:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-01T02:22:53.557 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:53.558 INFO:teuthology.orchestra.run.mira064:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:22:53.672 INFO:tasks.ceph:Setting up mon nodes... 2019-10-01T02:22:53.672 INFO:tasks.ceph:Setting up mgr nodes... 2019-10-01T02:22:53.672 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:53.672 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-10-01T02:22:54.190 INFO:teuthology.orchestra.run.mira064.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-10-01T02:22:54.197 INFO:tasks.ceph:Setting up mds nodes... 2019-10-01T02:22:54.197 INFO:tasks.ceph_client:Setting up client nodes... 2019-10-01T02:22:54.198 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:22:54.198 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-10-01T02:22:54.406 INFO:teuthology.orchestra.run.mira064.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-10-01T02:22:54.459 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-10-01T02:22:54.460 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@mira063.front.sepia.ceph.com'): {'osd.1': '/dev/sdf', 'osd.0': '/dev/sdh', 'osd.2': '/dev/sdg'}, Remote(name='ubuntu@mira064.front.sepia.ceph.com'): {'osd.3': '/dev/sdh', 'osd.5': '/dev/sdg', 'osd.4': '/dev/sdf'}} 2019-10-01T02:22:54.460 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:54.460 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-10-01T02:22:54.623 INFO:tasks.ceph:{'osd.1': '/dev/sdf', 'osd.0': '/dev/sdh', 'osd.2': '/dev/sdg'} 2019-10-01T02:22:54.623 INFO:tasks.ceph:{} 2019-10-01T02:22:54.623 INFO:tasks.ceph:osd.0 2019-10-01T02:22:54.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdh on ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:22:54.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:54.624 INFO:teuthology.orchestra.run.mira063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdh 2019-10-01T02:22:59.633 INFO:teuthology.orchestra.run.mira063.stdout:meta-data=/dev/sdh isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:22:59.633 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:22:59.634 INFO:teuthology.orchestra.run.mira063.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:22:59.634 INFO:teuthology.orchestra.run.mira063.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:22:59.634 INFO:teuthology.orchestra.run.mira063.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:22:59.635 INFO:teuthology.orchestra.run.mira063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:22:59.635 INFO:teuthology.orchestra.run.mira063.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:22:59.635 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:22:59.635 INFO:teuthology.orchestra.run.mira063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:22:59.638 INFO:tasks.ceph:mount /dev/sdh on ubuntu@mira063.front.sepia.ceph.com -o noatime 2019-10-01T02:22:59.638 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:22:59.638 INFO:teuthology.orchestra.run.mira063:> sudo mount -t xfs -o noatime /dev/sdh /var/lib/ceph/osd/ceph-0 2019-10-01T02:23:00.121 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:00.122 INFO:teuthology.orchestra.run.mira063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-10-01T02:23:00.279 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:00.279 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-10-01T02:23:00.436 INFO:tasks.ceph:{'osd.1': '/dev/sdf', 'osd.0': '/dev/sdh', 'osd.2': '/dev/sdg'} 2019-10-01T02:23:00.437 INFO:tasks.ceph:{} 2019-10-01T02:23:00.437 INFO:tasks.ceph:osd.1 2019-10-01T02:23:00.437 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdf on ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:23:00.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:00.437 INFO:teuthology.orchestra.run.mira063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdf 2019-10-01T02:23:11.245 INFO:teuthology.orchestra.run.mira063.stdout:meta-data=/dev/sdf isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:23:11.245 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:23:11.246 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:23:11.247 INFO:teuthology.orchestra.run.mira063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:23:11.249 INFO:tasks.ceph:mount /dev/sdf on ubuntu@mira063.front.sepia.ceph.com -o noatime 2019-10-01T02:23:11.249 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:11.249 INFO:teuthology.orchestra.run.mira063:> sudo mount -t xfs -o noatime /dev/sdf /var/lib/ceph/osd/ceph-1 2019-10-01T02:23:11.607 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:11.607 INFO:teuthology.orchestra.run.mira063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-10-01T02:23:11.768 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:11.769 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-10-01T02:23:11.915 INFO:tasks.ceph:{'osd.1': '/dev/sdf', 'osd.0': '/dev/sdh', 'osd.2': '/dev/sdg'} 2019-10-01T02:23:11.915 INFO:tasks.ceph:{} 2019-10-01T02:23:11.915 INFO:tasks.ceph:osd.2 2019-10-01T02:23:11.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdg on ubuntu@mira063.front.sepia.ceph.com 2019-10-01T02:23:11.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:11.916 INFO:teuthology.orchestra.run.mira063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdg 2019-10-01T02:23:15.556 INFO:teuthology.orchestra.run.mira063.stdout:meta-data=/dev/sdg isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:23:15.557 INFO:teuthology.orchestra.run.mira063.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:23:15.558 INFO:teuthology.orchestra.run.mira063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:23:15.558 INFO:teuthology.orchestra.run.mira063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:23:15.560 INFO:tasks.ceph:mount /dev/sdg on ubuntu@mira063.front.sepia.ceph.com -o noatime 2019-10-01T02:23:15.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:15.561 INFO:teuthology.orchestra.run.mira063:> sudo mount -t xfs -o noatime /dev/sdg /var/lib/ceph/osd/ceph-2 2019-10-01T02:23:16.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:16.183 INFO:teuthology.orchestra.run.mira063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-10-01T02:23:16.354 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:16.355 INFO:teuthology.orchestra.run.mira063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:16.603 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.597+0000 7fd3690cabc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-10-01T02:23:16.644 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.637+0000 7fd3690cabc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-10-01T02:23:16.644 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.637+0000 7fd3690cabc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-01T02:23:16.644 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.637+0000 7fd3690cabc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-01T02:23:16.644 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.638+0000 7fd3690cabc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-10-01T02:23:16.661 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:16.655+0000 7fd3690cabc0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-10-01T02:23:18.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:18.986 INFO:teuthology.orchestra.run.mira063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-10-01T02:23:19.151 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:19.152 INFO:teuthology.orchestra.run.mira063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:19.376 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.370+0000 7f7dcec45bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-10-01T02:23:19.377 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.371+0000 7f7dcec45bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-10-01T02:23:19.377 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.371+0000 7f7dcec45bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-01T02:23:19.377 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.371+0000 7f7dcec45bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-01T02:23:19.377 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.371+0000 7f7dcec45bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-10-01T02:23:19.379 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:19.373+0000 7f7dcec45bc0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-10-01T02:23:21.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:21.700 INFO:teuthology.orchestra.run.mira063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-10-01T02:23:21.860 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:21.860 INFO:teuthology.orchestra.run.mira063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:22.069 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.063+0000 7f1271941bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-10-01T02:23:22.069 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.063+0000 7f1271941bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-10-01T02:23:22.070 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.063+0000 7f1271941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-01T02:23:22.070 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.063+0000 7f1271941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-01T02:23:22.070 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.063+0000 7f1271941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-10-01T02:23:22.076 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:23:22.065+0000 7f1271941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-10-01T02:23:24.400 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:24.401 INFO:teuthology.orchestra.run.mira063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-10-01T02:23:24.573 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:24.573 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-10-01T02:23:24.734 INFO:tasks.ceph:{'osd.3': '/dev/sdh', 'osd.5': '/dev/sdg', 'osd.4': '/dev/sdf'} 2019-10-01T02:23:24.734 INFO:tasks.ceph:{} 2019-10-01T02:23:24.734 INFO:tasks.ceph:osd.3 2019-10-01T02:23:24.734 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdh on ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:23:24.734 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:24.735 INFO:teuthology.orchestra.run.mira064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdh 2019-10-01T02:23:29.119 INFO:teuthology.orchestra.run.mira064.stdout:meta-data=/dev/sdh isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:23:29.119 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:23:29.119 INFO:teuthology.orchestra.run.mira064.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:23:29.120 INFO:teuthology.orchestra.run.mira064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:23:29.123 INFO:tasks.ceph:mount /dev/sdh on ubuntu@mira064.front.sepia.ceph.com -o noatime 2019-10-01T02:23:29.123 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:29.123 INFO:teuthology.orchestra.run.mira064:> sudo mount -t xfs -o noatime /dev/sdh /var/lib/ceph/osd/ceph-3 2019-10-01T02:23:29.600 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:29.600 INFO:teuthology.orchestra.run.mira064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-10-01T02:23:29.778 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:29.779 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-10-01T02:23:29.933 INFO:tasks.ceph:{'osd.3': '/dev/sdh', 'osd.5': '/dev/sdg', 'osd.4': '/dev/sdf'} 2019-10-01T02:23:29.933 INFO:tasks.ceph:{} 2019-10-01T02:23:29.934 INFO:tasks.ceph:osd.4 2019-10-01T02:23:29.934 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdf on ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:23:29.934 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:29.934 INFO:teuthology.orchestra.run.mira064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdf 2019-10-01T02:23:34.204 INFO:teuthology.orchestra.run.mira064.stdout:meta-data=/dev/sdf isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:23:34.204 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:23:34.204 INFO:teuthology.orchestra.run.mira064.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:23:34.204 INFO:teuthology.orchestra.run.mira064.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:23:34.205 INFO:teuthology.orchestra.run.mira064.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:23:34.205 INFO:teuthology.orchestra.run.mira064.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:23:34.205 INFO:teuthology.orchestra.run.mira064.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:23:34.205 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:23:34.205 INFO:teuthology.orchestra.run.mira064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:23:34.207 INFO:tasks.ceph:mount /dev/sdf on ubuntu@mira064.front.sepia.ceph.com -o noatime 2019-10-01T02:23:34.207 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:34.208 INFO:teuthology.orchestra.run.mira064:> sudo mount -t xfs -o noatime /dev/sdf /var/lib/ceph/osd/ceph-4 2019-10-01T02:23:34.569 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:34.569 INFO:teuthology.orchestra.run.mira064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-10-01T02:23:34.735 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:34.735 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-10-01T02:23:34.889 INFO:tasks.ceph:{'osd.3': '/dev/sdh', 'osd.5': '/dev/sdg', 'osd.4': '/dev/sdf'} 2019-10-01T02:23:34.890 INFO:tasks.ceph:{} 2019-10-01T02:23:34.890 INFO:tasks.ceph:osd.5 2019-10-01T02:23:34.890 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/sdg on ubuntu@mira064.front.sepia.ceph.com 2019-10-01T02:23:34.890 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:34.890 INFO:teuthology.orchestra.run.mira064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/sdg 2019-10-01T02:23:39.210 INFO:teuthology.orchestra.run.mira064.stdout:meta-data=/dev/sdg isize=2048 agcount=4, agsize=61047662 blks 2019-10-01T02:23:39.210 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout: = crc=1 finobt=0, sparse=0 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout:data = bsize=4096 blocks=244190646, imaxpct=25 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout: = sunit=0 swidth=0 blks 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout:log =internal log bsize=4096 blocks=119233, version=2 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-01T02:23:39.211 INFO:teuthology.orchestra.run.mira064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-01T02:23:39.213 INFO:tasks.ceph:mount /dev/sdg on ubuntu@mira064.front.sepia.ceph.com -o noatime 2019-10-01T02:23:39.214 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:39.214 INFO:teuthology.orchestra.run.mira064:> sudo mount -t xfs -o noatime /dev/sdg /var/lib/ceph/osd/ceph-5 2019-10-01T02:23:39.586 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:39.586 INFO:teuthology.orchestra.run.mira064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-10-01T02:23:39.754 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:39.755 INFO:teuthology.orchestra.run.mira064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:39.986 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:39.983+0000 7f6ccebb7bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-10-01T02:23:40.000 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:39.996+0000 7f6ccebb7bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-10-01T02:23:40.000 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:39.996+0000 7f6ccebb7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-01T02:23:40.000 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:39.996+0000 7f6ccebb7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-01T02:23:40.001 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:39.996+0000 7f6ccebb7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-10-01T02:23:40.003 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:40.000+0000 7f6ccebb7bc0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-10-01T02:23:42.321 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:42.321 INFO:teuthology.orchestra.run.mira064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-10-01T02:23:42.486 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:42.487 INFO:teuthology.orchestra.run.mira064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:42.698 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.695+0000 7fccf085cbc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-10-01T02:23:42.713 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.708+0000 7fccf085cbc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-10-01T02:23:42.713 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.708+0000 7fccf085cbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-01T02:23:42.713 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.709+0000 7fccf085cbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-01T02:23:42.713 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.709+0000 7fccf085cbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-10-01T02:23:42.715 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:42.711+0000 7fccf085cbc0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-10-01T02:23:45.043 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:45.043 INFO:teuthology.orchestra.run.mira064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-10-01T02:23:45.208 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:45.208 INFO:teuthology.orchestra.run.mira064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:45.410 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.406+0000 7f1b525d5bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-10-01T02:23:45.411 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.407+0000 7f1b525d5bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-10-01T02:23:45.411 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.407+0000 7f1b525d5bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-01T02:23:45.412 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.407+0000 7f1b525d5bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-01T02:23:45.412 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.407+0000 7f1b525d5bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-10-01T02:23:45.412 INFO:teuthology.orchestra.run.mira064.stderr:2019-10-01T02:23:45.409+0000 7f1b525d5bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-10-01T02:23:47.740 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:47.740 INFO:teuthology.orchestra.run.mira064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-10-01T02:23:47.904 INFO:tasks.ceph:Reading keys from all nodes... 2019-10-01T02:23:47.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:47.905 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:48.017 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.27XbFz9yWq 2019-10-01T02:23:48.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.017 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.27XbFz9yWq 2019-10-01T02:23:48.169 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.169 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.27XbFz9yWq 2019-10-01T02:23:48.416 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.27XbFz9yWq is 56B 2019-10-01T02:23:48.539 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.539 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.27XbFz9yWq 2019-10-01T02:23:48.653 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.653 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:48.759 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.ciS5tUF7Pv 2019-10-01T02:23:48.759 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.760 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.ciS5tUF7Pv 2019-10-01T02:23:48.909 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:48.909 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.ciS5tUF7Pv 2019-10-01T02:23:49.185 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.ciS5tUF7Pv is 56B 2019-10-01T02:23:49.298 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:49.298 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.ciS5tUF7Pv 2019-10-01T02:23:49.400 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:49.401 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:49.500 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.3sBJayTvZu 2019-10-01T02:23:49.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:49.500 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.3sBJayTvZu 2019-10-01T02:23:49.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:49.646 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.3sBJayTvZu 2019-10-01T02:23:49.904 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.3sBJayTvZu is 56B 2019-10-01T02:23:50.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:50.017 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.3sBJayTvZu 2019-10-01T02:23:50.114 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:50.114 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:23:50.230 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.20RxipCXfp 2019-10-01T02:23:50.231 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:50.231 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.20RxipCXfp 2019-10-01T02:23:50.390 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:50.390 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.20RxipCXfp 2019-10-01T02:23:50.651 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.20RxipCXfp is 56B 2019-10-01T02:23:50.775 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:50.775 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.20RxipCXfp 2019-10-01T02:23:50.890 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:50.890 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:23:51.002 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.v9a78htXMl 2019-10-01T02:23:51.003 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.003 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.v9a78htXMl 2019-10-01T02:23:51.157 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.157 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.v9a78htXMl 2019-10-01T02:23:51.412 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.v9a78htXMl is 56B 2019-10-01T02:23:51.519 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.519 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.v9a78htXMl 2019-10-01T02:23:51.611 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.611 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:23:51.704 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.Oer3SW2nsN 2019-10-01T02:23:51.704 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.705 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.Oer3SW2nsN 2019-10-01T02:23:51.848 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:51.849 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.Oer3SW2nsN 2019-10-01T02:23:52.121 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.Oer3SW2nsN is 56B 2019-10-01T02:23:52.233 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:52.233 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.Oer3SW2nsN 2019-10-01T02:23:52.346 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:52.347 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:23:52.461 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.V7jGmPiHLY 2019-10-01T02:23:52.461 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:52.461 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.V7jGmPiHLY 2019-10-01T02:23:52.622 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:52.622 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.V7jGmPiHLY 2019-10-01T02:23:52.904 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.V7jGmPiHLY is 56B 2019-10-01T02:23:53.015 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:53.016 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.V7jGmPiHLY 2019-10-01T02:23:53.235 DEBUG:teuthology.orchestra.remote:mira064:/etc/ceph/ceph.client.0.keyring is 59B 2019-10-01T02:23:53.342 INFO:tasks.ceph:Adding keys to all mons... 2019-10-01T02:23:53.342 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:53.342 INFO:teuthology.orchestra.run.mira063:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-01T02:23:53.347 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:53.348 INFO:teuthology.orchestra.run.mira064:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout:[osd.0] 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCUuJJdw2SgIxAA5+6dUyh9wWgFkT1Q1ytx3w== 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout:[osd.1] 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCXuJJd0+8UFhAAmGFpz9ws+OhzIZh0UtCTxw== 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout:[osd.2] 2019-10-01T02:23:53.489 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCauJJdqW/IAxAACNRqO3SbW33IIv664ZgeFw== 2019-10-01T02:23:53.491 INFO:teuthology.orchestra.run.mira064.stdout:[mgr.x] 2019-10-01T02:23:53.491 INFO:teuthology.orchestra.run.mira064.stdout: key = AQB+uJJdb2csCxAA8Rd1i4junJT2jmR1/sltkw== 2019-10-01T02:23:53.491 INFO:teuthology.orchestra.run.mira064.stdout:[osd.3] 2019-10-01T02:23:53.491 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCruJJdpbynOhAAt7AjppIzVVDq/8lNhwQWPA== 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout:[osd.4] 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCuuJJdhp2AKRAAl/ht0KF05Rty7K5esFDGcQ== 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout:[osd.5] 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout: key = AQCxuJJduyVNGBAADyPCzsdKLeMK8uP4h8589Q== 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout:[client.0] 2019-10-01T02:23:53.492 INFO:teuthology.orchestra.run.mira064.stdout: key = AQB+uJJdrakYGBAAKTSlQ3Wo2bkBod4eAxU1IQ== 2019-10-01T02:23:53.500 INFO:teuthology.orchestra.run.mira063.stdout:[osd.0] 2019-10-01T02:23:53.500 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCUuJJdw2SgIxAA5+6dUyh9wWgFkT1Q1ytx3w== 2019-10-01T02:23:53.500 INFO:teuthology.orchestra.run.mira063.stdout:[osd.1] 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCXuJJd0+8UFhAAmGFpz9ws+OhzIZh0UtCTxw== 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout:[osd.2] 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCauJJdqW/IAxAACNRqO3SbW33IIv664ZgeFw== 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout:[mgr.x] 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout: key = AQB+uJJdb2csCxAA8Rd1i4junJT2jmR1/sltkw== 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout:[osd.3] 2019-10-01T02:23:53.501 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCruJJdpbynOhAAt7AjppIzVVDq/8lNhwQWPA== 2019-10-01T02:23:53.502 INFO:teuthology.orchestra.run.mira063.stdout:[osd.4] 2019-10-01T02:23:53.502 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCuuJJdhp2AKRAAl/ht0KF05Rty7K5esFDGcQ== 2019-10-01T02:23:53.502 INFO:teuthology.orchestra.run.mira063.stdout:[osd.5] 2019-10-01T02:23:53.502 INFO:teuthology.orchestra.run.mira063.stdout: key = AQCxuJJduyVNGBAADyPCzsdKLeMK8uP4h8589Q== 2019-10-01T02:23:53.502 INFO:teuthology.orchestra.run.mira063.stdout:[client.0] 2019-10-01T02:23:53.503 INFO:teuthology.orchestra.run.mira063.stdout: key = AQB+uJJdrakYGBAAKTSlQ3Wo2bkBod4eAxU1IQ== 2019-10-01T02:23:53.503 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:53.504 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:53.509 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:53.509 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:53.743 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:53.743 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:53.748 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:53.748 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:53.972 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:53.973 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:53.978 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:53.978 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:54.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-01T02:23:54.225 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:54.225 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-01T02:23:54.464 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:54.465 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.470 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:54.470 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:54.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.706 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:54.706 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.948 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:54.948 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:54.954 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:54.954 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-01T02:23:55.190 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:55.191 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-01T02:23:55.196 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:55.196 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-01T02:23:55.442 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-10-01T02:23:55.442 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:55.442 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-10-01T02:23:55.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:55.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-01T02:23:55.855 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:55.856 INFO:teuthology.orchestra.run.mira063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-10-01T02:23:56.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:56.016 INFO:teuthology.orchestra.run.mira063:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-10-01T02:23:56.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:56.178 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-01T02:23:56.440 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:56.441 INFO:teuthology.orchestra.run.mira063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-10-01T02:23:56.605 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:56.605 INFO:teuthology.orchestra.run.mira064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-10-01T02:23:56.771 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:56.771 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-01T02:23:57.029 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:57.029 INFO:teuthology.orchestra.run.mira064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-10-01T02:23:57.181 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:57.182 INFO:teuthology.orchestra.run.mira063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:57.187 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:57.188 INFO:teuthology.orchestra.run.mira064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-01T02:23:57.300 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-10-01T02:23:57.300 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:57.300 INFO:teuthology.orchestra.run.mira063:> which systemctl 2019-10-01T02:23:57.416 INFO:teuthology.orchestra.run.mira063.stdout:/usr/bin/systemctl 2019-10-01T02:23:57.416 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:23:57.416 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:57.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:23:57.422 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:23:57.423 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T02:23:57.423 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:57.423 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T02:23:57.427 INFO:tasks.ceph.mon.c:Started 2019-10-01T02:23:57.428 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:57.428 INFO:teuthology.orchestra.run.mira064:> which systemctl 2019-10-01T02:23:57.537 INFO:teuthology.orchestra.run.mira064.stdout:/usr/bin/systemctl 2019-10-01T02:23:57.537 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:23:57.538 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:57.538 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:23:57.543 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:23:57.543 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-10-01T02:23:57.544 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-10-01T02:23:57.544 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:23:57.544 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-10-01T02:23:57.549 INFO:tasks.ceph.mgr.x:Started 2019-10-01T02:23:57.549 INFO:tasks.ceph:Setting crush tunables to default 2019-10-01T02:23:57.549 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:57.549 INFO:teuthology.orchestra.run.mira063:> sudo ceph --cluster ceph osd crush tunables default 2019-10-01T02:23:58.334 INFO:teuthology.orchestra.run.mira063.stderr:adjusted tunables profile to default 2019-10-01T02:23:58.355 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-10-01T02:23:58.355 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:58.355 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:58.471 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.aj921isnQs 2019-10-01T02:23:58.472 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:58.472 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.aj921isnQs 2019-10-01T02:23:58.632 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:58.632 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.aj921isnQs 2019-10-01T02:23:58.926 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.aj921isnQs is 37B 2019-10-01T02:23:59.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.047 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.aj921isnQs 2019-10-01T02:23:59.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.150 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:59.245 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.AXIOJ67YOy 2019-10-01T02:23:59.246 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.246 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.AXIOJ67YOy 2019-10-01T02:23:59.406 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.406 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.AXIOJ67YOy 2019-10-01T02:23:59.663 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.AXIOJ67YOy is 37B 2019-10-01T02:23:59.772 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.772 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.AXIOJ67YOy 2019-10-01T02:23:59.871 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.871 INFO:teuthology.orchestra.run.mira063:> mktemp 2019-10-01T02:23:59.982 INFO:teuthology.orchestra.run.mira063.stdout:/tmp/tmp.mocIMT066y 2019-10-01T02:23:59.983 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:23:59.984 INFO:teuthology.orchestra.run.mira063:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.mocIMT066y 2019-10-01T02:24:00.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:00.133 INFO:teuthology.orchestra.run.mira063:> sudo chmod 0666 /tmp/tmp.mocIMT066y 2019-10-01T02:24:00.384 DEBUG:teuthology.orchestra.remote:mira063:/tmp/tmp.mocIMT066y is 37B 2019-10-01T02:24:00.490 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:00.490 INFO:teuthology.orchestra.run.mira063:> rm -fr /tmp/tmp.mocIMT066y 2019-10-01T02:24:00.604 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:00.604 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:24:00.677 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.VeUKsrz7Xg 2019-10-01T02:24:00.677 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:00.677 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.VeUKsrz7Xg 2019-10-01T02:24:00.769 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:00.769 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.VeUKsrz7Xg 2019-10-01T02:24:01.045 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.VeUKsrz7Xg is 37B 2019-10-01T02:24:01.156 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.157 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.VeUKsrz7Xg 2019-10-01T02:24:01.255 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.255 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:24:01.349 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.ZRX52edHzL 2019-10-01T02:24:01.350 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.350 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.ZRX52edHzL 2019-10-01T02:24:01.486 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.486 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.ZRX52edHzL 2019-10-01T02:24:01.730 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.ZRX52edHzL is 37B 2019-10-01T02:24:01.834 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.834 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.ZRX52edHzL 2019-10-01T02:24:01.931 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:01.931 INFO:teuthology.orchestra.run.mira064:> mktemp 2019-10-01T02:24:02.041 INFO:teuthology.orchestra.run.mira064.stdout:/tmp/tmp.2qAJTXIAgy 2019-10-01T02:24:02.042 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:02.042 INFO:teuthology.orchestra.run.mira064:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.2qAJTXIAgy 2019-10-01T02:24:02.194 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:02.194 INFO:teuthology.orchestra.run.mira064:> sudo chmod 0666 /tmp/tmp.2qAJTXIAgy 2019-10-01T02:24:02.456 DEBUG:teuthology.orchestra.remote:mira064:/tmp/tmp.2qAJTXIAgy is 37B 2019-10-01T02:24:02.561 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:02.561 INFO:teuthology.orchestra.run.mira064:> rm -fr /tmp/tmp.2qAJTXIAgy 2019-10-01T02:24:02.664 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:02.665 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new eead52b7-7821-4283-b7c2-3ea9c4b1b9d2 0 2019-10-01T02:24:03.596 INFO:teuthology.orchestra.run.mira064.stdout:0 2019-10-01T02:24:03.626 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:03.626 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new ab58538f-ffa8-4f9b-85c9-bfbcebc0959d 1 2019-10-01T02:24:04.553 INFO:teuthology.orchestra.run.mira064.stdout:1 2019-10-01T02:24:04.575 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:04.575 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new 243ceb1b-964e-4226-83a5-5bcaeb66e71e 2 2019-10-01T02:24:05.511 INFO:teuthology.orchestra.run.mira064.stdout:2 2019-10-01T02:24:05.529 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:05.529 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new d13ee4b5-26e5-4b5c-95ed-49770e5613f7 3 2019-10-01T02:24:06.428 INFO:teuthology.orchestra.run.mira064.stdout:3 2019-10-01T02:24:06.451 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:06.451 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new 91591ad5-2fc6-49b3-b9dd-4b9ac6019691 4 2019-10-01T02:24:07.399 INFO:teuthology.orchestra.run.mira064.stdout:4 2019-10-01T02:24:07.428 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:07.428 INFO:teuthology.orchestra.run.mira064:> sudo ceph --cluster ceph osd new 23d48522-d36c-4fe3-8c49-d34479580806 5 2019-10-01T02:24:08.347 INFO:teuthology.orchestra.run.mira064.stdout:5 2019-10-01T02:24:08.365 INFO:tasks.ceph.osd.0:Restarting daemon 2019-10-01T02:24:08.365 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:08.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-10-01T02:24:08.374 INFO:tasks.ceph.osd.0:Started 2019-10-01T02:24:08.374 INFO:tasks.ceph.osd.1:Restarting daemon 2019-10-01T02:24:08.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:08.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-10-01T02:24:08.379 INFO:tasks.ceph.osd.1:Started 2019-10-01T02:24:08.379 INFO:tasks.ceph.osd.2:Restarting daemon 2019-10-01T02:24:08.380 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:08.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-10-01T02:24:08.384 INFO:tasks.ceph.osd.2:Started 2019-10-01T02:24:08.385 INFO:tasks.ceph.osd.3:Restarting daemon 2019-10-01T02:24:08.385 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:08.385 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-10-01T02:24:08.390 INFO:tasks.ceph.osd.3:Started 2019-10-01T02:24:08.390 INFO:tasks.ceph.osd.4:Restarting daemon 2019-10-01T02:24:08.390 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:08.391 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-10-01T02:24:08.395 INFO:tasks.ceph.osd.4:Started 2019-10-01T02:24:08.396 INFO:tasks.ceph.osd.5:Restarting daemon 2019-10-01T02:24:08.396 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:08.396 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-10-01T02:24:08.401 INFO:tasks.ceph.osd.5:Started 2019-10-01T02:24:08.401 INFO:tasks.ceph:Waiting for OSDs to come up 2019-10-01T02:24:09.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:09.008 INFO:teuthology.orchestra.run.mira063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:09.107 INFO:tasks.ceph.osd.4.mira064.stderr:2019-10-01T02:24:09.103+0000 7f94295d6bc0 -1 Falling back to public interface 2019-10-01T02:24:09.111 INFO:tasks.ceph.osd.3.mira064.stderr:2019-10-01T02:24:09.107+0000 7fca09496bc0 -1 Falling back to public interface 2019-10-01T02:24:09.114 INFO:tasks.ceph.osd.5.mira064.stderr:2019-10-01T02:24:09.110+0000 7f4c835fabc0 -1 Falling back to public interface 2019-10-01T02:24:09.147 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:24:09.140+0000 7fed66c9fbc0 -1 Falling back to public interface 2019-10-01T02:24:09.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T02:24:09.141+0000 7f10c0692bc0 -1 Falling back to public interface 2019-10-01T02:24:09.149 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:24:09.141+0000 7fed61567700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2019-10-01T02:24:09.149 INFO:tasks.ceph.osd.2.mira063.stderr:2019-10-01T02:24:09.142+0000 7f6424d31bc0 -1 Falling back to public interface 2019-10-01T02:24:09.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T02:24:09.143+0000 7f10baf5a700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2019-10-01T02:24:09.153 INFO:tasks.ceph.osd.2.mira063.stderr:2019-10-01T02:24:09.146+0000 7f641f5f9700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6817/0: (98) Address already in use 2019-10-01T02:24:09.935 INFO:teuthology.misc.health.mira063.stdout: 2019-10-01T02:24:09.935 INFO:teuthology.misc.health.mira063.stdout:{"epoch":8,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:08.330608+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:09.953 DEBUG:teuthology.misc:0 of 6 OSDs are up 2019-10-01T02:24:10.188 INFO:tasks.ceph.osd.4.mira064.stderr:2019-10-01T02:24:10.183+0000 7f94295d6bc0 -1 osd.4 0 log_to_monitors {default=true} 2019-10-01T02:24:10.196 INFO:tasks.ceph.osd.5.mira064.stderr:2019-10-01T02:24:10.192+0000 7f4c835fabc0 -1 osd.5 0 log_to_monitors {default=true} 2019-10-01T02:24:10.196 INFO:tasks.ceph.osd.3.mira064.stderr:2019-10-01T02:24:10.193+0000 7fca09496bc0 -1 osd.3 0 log_to_monitors {default=true} 2019-10-01T02:24:10.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T02:24:10.205+0000 7f10c0692bc0 -1 osd.0 0 log_to_monitors {default=true} 2019-10-01T02:24:10.217 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:24:10.205+0000 7fed66c9fbc0 -1 osd.1 0 log_to_monitors {default=true} 2019-10-01T02:24:10.227 INFO:tasks.ceph.osd.2.mira063.stderr:2019-10-01T02:24:10.220+0000 7f6424d31bc0 -1 osd.2 0 log_to_monitors {default=true} 2019-10-01T02:24:12.022 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:24:12.014+0000 7fed4caa9700 -1 osd.1 0 waiting for initial osdmap 2019-10-01T02:24:12.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T02:24:12.015+0000 7f10a649c700 -1 osd.0 0 waiting for initial osdmap 2019-10-01T02:24:12.023 INFO:tasks.ceph.osd.2.mira063.stderr:2019-10-01T02:24:12.016+0000 7f640ab3b700 -1 osd.2 0 waiting for initial osdmap 2019-10-01T02:24:12.025 INFO:tasks.ceph.osd.4.mira064.stderr:2019-10-01T02:24:12.022+0000 7f940f3e0700 -1 osd.4 0 waiting for initial osdmap 2019-10-01T02:24:12.025 INFO:tasks.ceph.osd.5.mira064.stderr:2019-10-01T02:24:12.022+0000 7f4c69404700 -1 osd.5 0 waiting for initial osdmap 2019-10-01T02:24:12.026 INFO:tasks.ceph.osd.3.mira064.stderr:2019-10-01T02:24:12.022+0000 7fc9ef2a0700 -1 osd.3 0 waiting for initial osdmap 2019-10-01T02:24:16.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:16.558 INFO:teuthology.orchestra.run.mira063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:17.513 INFO:teuthology.misc.health.mira063.stdout: 2019-10-01T02:24:17.513 INFO:teuthology.misc.health.mira063.stdout:{"epoch":12,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:14.038867+0000","last_up_change":"2019-10-01T02:24:13.019960+0000","last_in_change":"2019-10-01T02:24:13.019960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6804","nonce":7027},{"type":"v1","addr":"172.21.6.124:6806","nonce":7027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6810","nonce":7027},{"type":"v1","addr":"172.21.6.124:6813","nonce":7027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6819","nonce":7027},{"type":"v1","addr":"172.21.6.124:6821","nonce":7027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6816","nonce":7027},{"type":"v1","addr":"172.21.6.124:6817","nonce":7027}]},"public_addr":"172.21.6.124:6806/7027","cluster_addr":"172.21.6.124:6813/7027","heartbeat_back_addr":"172.21.6.124:6821/7027","heartbeat_front_addr":"172.21.6.124:6817/7027","state":["exists","up"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6800","nonce":7026},{"type":"v1","addr":"172.21.6.124:6801","nonce":7026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6802","nonce":7026},{"type":"v1","addr":"172.21.6.124:6803","nonce":7026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6811","nonce":7026},{"type":"v1","addr":"172.21.6.124:6814","nonce":7026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6805","nonce":7026},{"type":"v1","addr":"172.21.6.124:6808","nonce":7026}]},"public_addr":"172.21.6.124:6801/7026","cluster_addr":"172.21.6.124:6803/7026","heartbeat_back_addr":"172.21.6.124:6814/7026","heartbeat_front_addr":"172.21.6.124:6808/7026","state":["exists","up"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6807","nonce":7028},{"type":"v1","addr":"172.21.6.124:6809","nonce":7028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6812","nonce":7028},{"type":"v1","addr":"172.21.6.124:6815","nonce":7028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6822","nonce":7028},{"type":"v1","addr":"172.21.6.124:6823","nonce":7028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6818","nonce":7028},{"type":"v1","addr":"172.21.6.124:6820","nonce":7028}]},"public_addr":"172.21.6.124:6809/7028","cluster_addr":"172.21.6.124:6815/7028","heartbeat_back_addr":"172.21.6.124:6823/7028","heartbeat_front_addr":"172.21.6.124:6820/7028","state":["exists","up"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6810","nonce":7131},{"type":"v1","addr":"172.21.6.120:6811","nonce":7131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6812","nonce":7131},{"type":"v1","addr":"172.21.6.120:6813","nonce":7131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6819","nonce":7131},{"type":"v1","addr":"172.21.6.120:6821","nonce":7131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6815","nonce":7131},{"type":"v1","addr":"172.21.6.120:6817","nonce":7131}]},"public_addr":"172.21.6.120:6811/7131","cluster_addr":"172.21.6.120:6813/7131","heartbeat_back_addr":"172.21.6.120:6821/7131","heartbeat_front_addr":"172.21.6.120:6817/7131","state":["exists","up"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6802","nonce":7132},{"type":"v1","addr":"172.21.6.120:6803","nonce":7132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6804","nonce":7132},{"type":"v1","addr":"172.21.6.120:6805","nonce":7132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6808","nonce":7132},{"type":"v1","addr":"172.21.6.120:6809","nonce":7132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6806","nonce":7132},{"type":"v1","addr":"172.21.6.120:6807","nonce":7132}]},"public_addr":"172.21.6.120:6803/7132","cluster_addr":"172.21.6.120:6805/7132","heartbeat_back_addr":"172.21.6.120:6809/7132","heartbeat_front_addr":"172.21.6.120:6807/7132","state":["exists","up"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6814","nonce":7133},{"type":"v1","addr":"172.21.6.120:6816","nonce":7133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6818","nonce":7133},{"type":"v1","addr":"172.21.6.120:6820","nonce":7133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6824","nonce":7133},{"type":"v1","addr":"172.21.6.120:6825","nonce":7133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6822","nonce":7133},{"type":"v1","addr":"172.21.6.120:6823","nonce":7133}]},"public_addr":"172.21.6.120:6816/7133","cluster_addr":"172.21.6.120:6820/7133","heartbeat_back_addr":"172.21.6.120:6825/7133","heartbeat_front_addr":"172.21.6.120:6823/7133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.194318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.256827+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.221457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.173318+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.143400+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.189943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:17.532 DEBUG:teuthology.misc:6 of 6 OSDs are up 2019-10-01T02:24:17.533 INFO:tasks.ceph:Creating RBD pool 2019-10-01T02:24:17.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:17.533 INFO:teuthology.orchestra.run.mira063:> sudo ceph --cluster ceph osd pool create rbd 8 2019-10-01T02:24:18.907 INFO:teuthology.orchestra.run.mira063.stderr:pool 'rbd' created 2019-10-01T02:24:18.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:18.938 INFO:teuthology.orchestra.run.mira063:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-10-01T02:24:20.893 INFO:teuthology.orchestra.run.mira063.stderr:enabled application 'rbd' on pool 'rbd' 2019-10-01T02:24:20.921 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-10-01T02:24:20.923 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:20.923 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:20.924 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-10-01T02:24:21.877 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:21.877 INFO:teuthology.orchestra.run.mira063.stdout:{"epoch":15,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:20.871939+0000","last_up_change":"2019-10-01T02:24:13.019960+0000","last_in_change":"2019-10-01T02:24:13.019960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-01T02:24:18.500383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6804","nonce":7027},{"type":"v1","addr":"172.21.6.124:6806","nonce":7027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6810","nonce":7027},{"type":"v1","addr":"172.21.6.124:6813","nonce":7027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6819","nonce":7027},{"type":"v1","addr":"172.21.6.124:6821","nonce":7027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6816","nonce":7027},{"type":"v1","addr":"172.21.6.124:6817","nonce":7027}]},"public_addr":"172.21.6.124:6806/7027","cluster_addr":"172.21.6.124:6813/7027","heartbeat_back_addr":"172.21.6.124:6821/7027","heartbeat_front_addr":"172.21.6.124:6817/7027","state":["exists","up"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6800","nonce":7026},{"type":"v1","addr":"172.21.6.124:6801","nonce":7026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6802","nonce":7026},{"type":"v1","addr":"172.21.6.124:6803","nonce":7026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6811","nonce":7026},{"type":"v1","addr":"172.21.6.124:6814","nonce":7026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6805","nonce":7026},{"type":"v1","addr":"172.21.6.124:6808","nonce":7026}]},"public_addr":"172.21.6.124:6801/7026","cluster_addr":"172.21.6.124:6803/7026","heartbeat_back_addr":"172.21.6.124:6814/7026","heartbeat_front_addr":"172.21.6.124:6808/7026","state":["exists","up"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6807","nonce":7028},{"type":"v1","addr":"172.21.6.124:6809","nonce":7028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6812","nonce":7028},{"type":"v1","addr":"172.21.6.124:6815","nonce":7028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6822","nonce":7028},{"type":"v1","addr":"172.21.6.124:6823","nonce":7028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6818","nonce":7028},{"type":"v1","addr":"172.21.6.124:6820","nonce":7028}]},"public_addr":"172.21.6.124:6809/7028","cluster_addr":"172.21.6.124:6815/7028","heartbeat_back_addr":"172.21.6.124:6823/7028","heartbeat_front_addr":"172.21.6.124:6820/7028","state":["exists","up"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6810","nonce":7131},{"type":"v1","addr":"172.21.6.120:6811","nonce":7131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6812","nonce":7131},{"type":"v1","addr":"172.21.6.120:6813","nonce":7131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6819","nonce":7131},{"type":"v1","addr":"172.21.6.120:6821","nonce":7131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6815","nonce":7131},{"type":"v1","addr":"172.21.6.120:6817","nonce":7131}]},"public_addr":"172.21.6.120:6811/7131","cluster_addr":"172.21.6.120:6813/7131","heartbeat_back_addr":"172.21.6.120:6821/7131","heartbeat_front_addr":"172.21.6.120:6817/7131","state":["exists","up"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6802","nonce":7132},{"type":"v1","addr":"172.21.6.120:6803","nonce":7132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6804","nonce":7132},{"type":"v1","addr":"172.21.6.120:6805","nonce":7132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6808","nonce":7132},{"type":"v1","addr":"172.21.6.120:6809","nonce":7132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6806","nonce":7132},{"type":"v1","addr":"172.21.6.120:6807","nonce":7132}]},"public_addr":"172.21.6.120:6803/7132","cluster_addr":"172.21.6.120:6805/7132","heartbeat_back_addr":"172.21.6.120:6809/7132","heartbeat_front_addr":"172.21.6.120:6807/7132","state":["exists","up"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6814","nonce":7133},{"type":"v1","addr":"172.21.6.120:6816","nonce":7133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6818","nonce":7133},{"type":"v1","addr":"172.21.6.120:6820","nonce":7133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6824","nonce":7133},{"type":"v1","addr":"172.21.6.120:6825","nonce":7133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6822","nonce":7133},{"type":"v1","addr":"172.21.6.120:6823","nonce":7133}]},"public_addr":"172.21.6.120:6816/7133","cluster_addr":"172.21.6.120:6820/7133","heartbeat_back_addr":"172.21.6.120:6825/7133","heartbeat_front_addr":"172.21.6.120:6823/7133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.194318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.256827+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.221457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.173318+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.143400+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.189943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:21.900 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-01T02:24:18.500383+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'15', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-01T02:24:21.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:21.900 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-01T02:24:22.814 INFO:teuthology.orchestra.run.mira063.stdout:pg_num: 8 2019-10-01T02:24:22.845 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-10-01T02:24:22.846 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-10-01T02:24:22.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:22.846 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-10-01T02:24:23.793 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:23.815 INFO:teuthology.orchestra.run.mira063.stdout:{"epoch":3,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6800","nonce":6112},{"type":"v1","addr":"172.21.6.120:6801","nonce":6112}]},"active_addr":"172.21.6.120:6801/6112","active_change":"2019-10-01T02:24:00.773839+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-10-01T02:24:23.817 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-10-01T02:24:24.423 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:24.423 INFO:teuthology.orchestra.run.mira063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:25.373 INFO:teuthology.misc.health.mira063.stdout: 2019-10-01T02:24:25.374 INFO:teuthology.misc.health.mira063.stdout:{"epoch":15,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:20.871939+0000","last_up_change":"2019-10-01T02:24:13.019960+0000","last_in_change":"2019-10-01T02:24:13.019960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-01T02:24:18.500383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6804","nonce":7027},{"type":"v1","addr":"172.21.6.124:6806","nonce":7027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6810","nonce":7027},{"type":"v1","addr":"172.21.6.124:6813","nonce":7027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6819","nonce":7027},{"type":"v1","addr":"172.21.6.124:6821","nonce":7027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6816","nonce":7027},{"type":"v1","addr":"172.21.6.124:6817","nonce":7027}]},"public_addr":"172.21.6.124:6806/7027","cluster_addr":"172.21.6.124:6813/7027","heartbeat_back_addr":"172.21.6.124:6821/7027","heartbeat_front_addr":"172.21.6.124:6817/7027","state":["exists","up"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6800","nonce":7026},{"type":"v1","addr":"172.21.6.124:6801","nonce":7026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6802","nonce":7026},{"type":"v1","addr":"172.21.6.124:6803","nonce":7026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6811","nonce":7026},{"type":"v1","addr":"172.21.6.124:6814","nonce":7026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6805","nonce":7026},{"type":"v1","addr":"172.21.6.124:6808","nonce":7026}]},"public_addr":"172.21.6.124:6801/7026","cluster_addr":"172.21.6.124:6803/7026","heartbeat_back_addr":"172.21.6.124:6814/7026","heartbeat_front_addr":"172.21.6.124:6808/7026","state":["exists","up"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6807","nonce":7028},{"type":"v1","addr":"172.21.6.124:6809","nonce":7028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6812","nonce":7028},{"type":"v1","addr":"172.21.6.124:6815","nonce":7028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6822","nonce":7028},{"type":"v1","addr":"172.21.6.124:6823","nonce":7028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6818","nonce":7028},{"type":"v1","addr":"172.21.6.124:6820","nonce":7028}]},"public_addr":"172.21.6.124:6809/7028","cluster_addr":"172.21.6.124:6815/7028","heartbeat_back_addr":"172.21.6.124:6823/7028","heartbeat_front_addr":"172.21.6.124:6820/7028","state":["exists","up"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6810","nonce":7131},{"type":"v1","addr":"172.21.6.120:6811","nonce":7131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6812","nonce":7131},{"type":"v1","addr":"172.21.6.120:6813","nonce":7131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6819","nonce":7131},{"type":"v1","addr":"172.21.6.120:6821","nonce":7131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6815","nonce":7131},{"type":"v1","addr":"172.21.6.120:6817","nonce":7131}]},"public_addr":"172.21.6.120:6811/7131","cluster_addr":"172.21.6.120:6813/7131","heartbeat_back_addr":"172.21.6.120:6821/7131","heartbeat_front_addr":"172.21.6.120:6817/7131","state":["exists","up"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6802","nonce":7132},{"type":"v1","addr":"172.21.6.120:6803","nonce":7132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6804","nonce":7132},{"type":"v1","addr":"172.21.6.120:6805","nonce":7132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6808","nonce":7132},{"type":"v1","addr":"172.21.6.120:6809","nonce":7132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6806","nonce":7132},{"type":"v1","addr":"172.21.6.120:6807","nonce":7132}]},"public_addr":"172.21.6.120:6803/7132","cluster_addr":"172.21.6.120:6805/7132","heartbeat_back_addr":"172.21.6.120:6809/7132","heartbeat_front_addr":"172.21.6.120:6807/7132","state":["exists","up"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6814","nonce":7133},{"type":"v1","addr":"172.21.6.120:6816","nonce":7133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6818","nonce":7133},{"type":"v1","addr":"172.21.6.120:6820","nonce":7133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6824","nonce":7133},{"type":"v1","addr":"172.21.6.120:6825","nonce":7133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6822","nonce":7133},{"type":"v1","addr":"172.21.6.120:6823","nonce":7133}]},"public_addr":"172.21.6.120:6816/7133","cluster_addr":"172.21.6.120:6820/7133","heartbeat_back_addr":"172.21.6.120:6825/7133","heartbeat_front_addr":"172.21.6.120:6823/7133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.194318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.256827+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.221457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.173318+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.143400+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.189943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:25.395 DEBUG:teuthology.misc:6 of 6 OSDs are up 2019-10-01T02:24:25.395 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:25.396 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:26.323 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:26.324 INFO:teuthology.orchestra.run.mira063.stdout:{"epoch":15,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:20.871939+0000","last_up_change":"2019-10-01T02:24:13.019960+0000","last_in_change":"2019-10-01T02:24:13.019960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-01T02:24:18.500383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6804","nonce":7027},{"type":"v1","addr":"172.21.6.124:6806","nonce":7027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6810","nonce":7027},{"type":"v1","addr":"172.21.6.124:6813","nonce":7027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6819","nonce":7027},{"type":"v1","addr":"172.21.6.124:6821","nonce":7027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6816","nonce":7027},{"type":"v1","addr":"172.21.6.124:6817","nonce":7027}]},"public_addr":"172.21.6.124:6806/7027","cluster_addr":"172.21.6.124:6813/7027","heartbeat_back_addr":"172.21.6.124:6821/7027","heartbeat_front_addr":"172.21.6.124:6817/7027","state":["exists","up"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6800","nonce":7026},{"type":"v1","addr":"172.21.6.124:6801","nonce":7026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6802","nonce":7026},{"type":"v1","addr":"172.21.6.124:6803","nonce":7026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6811","nonce":7026},{"type":"v1","addr":"172.21.6.124:6814","nonce":7026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6805","nonce":7026},{"type":"v1","addr":"172.21.6.124:6808","nonce":7026}]},"public_addr":"172.21.6.124:6801/7026","cluster_addr":"172.21.6.124:6803/7026","heartbeat_back_addr":"172.21.6.124:6814/7026","heartbeat_front_addr":"172.21.6.124:6808/7026","state":["exists","up"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6807","nonce":7028},{"type":"v1","addr":"172.21.6.124:6809","nonce":7028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6812","nonce":7028},{"type":"v1","addr":"172.21.6.124:6815","nonce":7028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6822","nonce":7028},{"type":"v1","addr":"172.21.6.124:6823","nonce":7028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6818","nonce":7028},{"type":"v1","addr":"172.21.6.124:6820","nonce":7028}]},"public_addr":"172.21.6.124:6809/7028","cluster_addr":"172.21.6.124:6815/7028","heartbeat_back_addr":"172.21.6.124:6823/7028","heartbeat_front_addr":"172.21.6.124:6820/7028","state":["exists","up"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6810","nonce":7131},{"type":"v1","addr":"172.21.6.120:6811","nonce":7131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6812","nonce":7131},{"type":"v1","addr":"172.21.6.120:6813","nonce":7131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6819","nonce":7131},{"type":"v1","addr":"172.21.6.120:6821","nonce":7131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6815","nonce":7131},{"type":"v1","addr":"172.21.6.120:6817","nonce":7131}]},"public_addr":"172.21.6.120:6811/7131","cluster_addr":"172.21.6.120:6813/7131","heartbeat_back_addr":"172.21.6.120:6821/7131","heartbeat_front_addr":"172.21.6.120:6817/7131","state":["exists","up"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6802","nonce":7132},{"type":"v1","addr":"172.21.6.120:6803","nonce":7132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6804","nonce":7132},{"type":"v1","addr":"172.21.6.120:6805","nonce":7132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6808","nonce":7132},{"type":"v1","addr":"172.21.6.120:6809","nonce":7132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6806","nonce":7132},{"type":"v1","addr":"172.21.6.120:6807","nonce":7132}]},"public_addr":"172.21.6.120:6803/7132","cluster_addr":"172.21.6.120:6805/7132","heartbeat_back_addr":"172.21.6.120:6809/7132","heartbeat_front_addr":"172.21.6.120:6807/7132","state":["exists","up"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6814","nonce":7133},{"type":"v1","addr":"172.21.6.120:6816","nonce":7133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6818","nonce":7133},{"type":"v1","addr":"172.21.6.120:6820","nonce":7133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6824","nonce":7133},{"type":"v1","addr":"172.21.6.120:6825","nonce":7133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6822","nonce":7133},{"type":"v1","addr":"172.21.6.120:6823","nonce":7133}]},"public_addr":"172.21.6.120:6816/7133","cluster_addr":"172.21.6.120:6820/7133","heartbeat_back_addr":"172.21.6.120:6825/7133","heartbeat_front_addr":"172.21.6.120:6823/7133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.194318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.256827+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.221457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.173318+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.143400+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.189943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:26.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:26.353 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-10-01T02:24:26.830 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:26.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:26.855 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-10-01T02:24:27.336 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:27.362 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:27.362 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-10-01T02:24:27.827 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:27.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:27.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-10-01T02:24:28.338 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:28.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:28.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-10-01T02:24:28.841 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:28.864 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:28.864 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-10-01T02:24:29.304 INFO:teuthology.orchestra.run.mira063.stdout:47244640263 2019-10-01T02:24:29.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:29.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-01T02:24:30.224 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:30.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.0 2019-10-01T02:24:30.247 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:30.247 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-01T02:24:31.177 INFO:teuthology.orchestra.run.mira063.stdout:47244640262 2019-10-01T02:24:31.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.1 2019-10-01T02:24:31.198 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:31.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-10-01T02:24:32.126 INFO:teuthology.orchestra.run.mira063.stdout:47244640263 2019-10-01T02:24:32.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640263 for osd.2 2019-10-01T02:24:32.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:32.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-10-01T02:24:33.154 INFO:teuthology.orchestra.run.mira063.stdout:47244640263 2019-10-01T02:24:33.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640263 for osd.3 2019-10-01T02:24:33.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:33.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-01T02:24:34.109 INFO:teuthology.orchestra.run.mira063.stdout:47244640263 2019-10-01T02:24:34.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640263 for osd.4 2019-10-01T02:24:34.131 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:34.132 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-10-01T02:24:35.053 INFO:teuthology.orchestra.run.mira063.stdout:47244640264 2019-10-01T02:24:35.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640263 got 47244640264 for osd.5 2019-10-01T02:24:35.075 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-10-01T02:24:35.075 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:35.075 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-01T02:24:36.029 INFO:teuthology.orchestra.run.mira063.stderr:dumped all 2019-10-01T02:24:36.029 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:36.051 INFO:teuthology.orchestra.run.mira063.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2019-10-01T02:24:34.795201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":6304128,"kb_used_data":12288,"kb_used_omap":0,"kb_used_meta":6291456,"kb_avail":559926912,"statfs":{"total":579820584960,"available":573365157888,"internally_reserved":6442450944,"allocated":12582912,"data_stored":243708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6442450944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884310+0000","last_change":"2019-10-01T02:24:19.883996+0000","last_active":"2019-10-01T02:24:19.884310+0000","last_peered":"2019-10-01T02:24:19.884310+0000","last_clean":"2019-10-01T02:24:19.884310+0000","last_became_active":"2019-10-01T02:24:19.883516+0000","last_became_peered":"2019-10-01T02:24:19.883516+0000","last_unstale":"2019-10-01T02:24:19.884310+0000","last_undegraded":"2019-10-01T02:24:19.884310+0000","last_fullsized":"2019-10-01T02:24:19.884310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881661+0000","last_change":"2019-10-01T02:24:19.880871+0000","last_active":"2019-10-01T02:24:19.881661+0000","last_peered":"2019-10-01T02:24:19.881661+0000","last_clean":"2019-10-01T02:24:19.881661+0000","last_became_active":"2019-10-01T02:24:19.879992+0000","last_became_peered":"2019-10-01T02:24:19.879992+0000","last_unstale":"2019-10-01T02:24:19.881661+0000","last_undegraded":"2019-10-01T02:24:19.881661+0000","last_fullsized":"2019-10-01T02:24:19.881661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884030+0000","last_change":"2019-10-01T02:24:19.883279+0000","last_active":"2019-10-01T02:24:19.884030+0000","last_peered":"2019-10-01T02:24:19.884030+0000","last_clean":"2019-10-01T02:24:19.884030+0000","last_became_active":"2019-10-01T02:24:19.882828+0000","last_became_peered":"2019-10-01T02:24:19.882828+0000","last_unstale":"2019-10-01T02:24:19.884030+0000","last_undegraded":"2019-10-01T02:24:19.884030+0000","last_fullsized":"2019-10-01T02:24:19.884030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.888642+0000","last_change":"2019-10-01T02:24:19.887717+0000","last_active":"2019-10-01T02:24:19.888642+0000","last_peered":"2019-10-01T02:24:19.888642+0000","last_clean":"2019-10-01T02:24:19.888642+0000","last_became_active":"2019-10-01T02:24:19.886121+0000","last_became_peered":"2019-10-01T02:24:19.886121+0000","last_unstale":"2019-10-01T02:24:19.888642+0000","last_undegraded":"2019-10-01T02:24:19.888642+0000","last_fullsized":"2019-10-01T02:24:19.888642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881438+0000","last_change":"2019-10-01T02:24:19.880892+0000","last_active":"2019-10-01T02:24:19.881438+0000","last_peered":"2019-10-01T02:24:19.881438+0000","last_clean":"2019-10-01T02:24:19.881438+0000","last_became_active":"2019-10-01T02:24:19.880369+0000","last_became_peered":"2019-10-01T02:24:19.880369+0000","last_unstale":"2019-10-01T02:24:19.881438+0000","last_undegraded":"2019-10-01T02:24:19.881438+0000","last_fullsized":"2019-10-01T02:24:19.881438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.882972+0000","last_change":"2019-10-01T02:24:19.882549+0000","last_active":"2019-10-01T02:24:19.882972+0000","last_peered":"2019-10-01T02:24:19.882972+0000","last_clean":"2019-10-01T02:24:19.882972+0000","last_became_active":"2019-10-01T02:24:19.882007+0000","last_became_peered":"2019-10-01T02:24:19.882007+0000","last_unstale":"2019-10-01T02:24:19.882972+0000","last_undegraded":"2019-10-01T02:24:19.882972+0000","last_fullsized":"2019-10-01T02:24:19.882972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.883411+0000","last_change":"2019-10-01T02:24:19.883094+0000","last_active":"2019-10-01T02:24:19.883411+0000","last_peered":"2019-10-01T02:24:19.883411+0000","last_clean":"2019-10-01T02:24:19.883411+0000","last_became_active":"2019-10-01T02:24:19.882560+0000","last_became_peered":"2019-10-01T02:24:19.882560+0000","last_unstale":"2019-10-01T02:24:19.883411+0000","last_undegraded":"2019-10-01T02:24:19.883411+0000","last_fullsized":"2019-10-01T02:24:19.883411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.886062+0000","last_change":"2019-10-01T02:24:19.884777+0000","last_active":"2019-10-01T02:24:19.886062+0000","last_peered":"2019-10-01T02:24:19.886062+0000","last_clean":"2019-10-01T02:24:19.886062+0000","last_became_active":"2019-10-01T02:24:19.883514+0000","last_became_peered":"2019-10-01T02:24:19.883514+0000","last_unstale":"2019-10-01T02:24:19.886062+0000","last_undegraded":"2019-10-01T02:24:19.886062+0000","last_fullsized":"2019-10-01T02:24:19.886062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.199}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.933}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.762}]}]},{"osd":4,"up_from":11,"seq":47244640264,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.262}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.860}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.117}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.719}]}]},{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.466}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.636}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.929}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.414}]}]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.427}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.348}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.865}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.966}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.884}]}]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.721}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.826}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.155}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.090}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.570}]}]},{"osd":3,"up_from":11,"seq":47244640264,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.982}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.775}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.892}]}]}]}} 2019-10-01T02:24:36.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:36.053 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-01T02:24:36.997 INFO:teuthology.orchestra.run.mira063.stderr:dumped all 2019-10-01T02:24:36.997 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:37.025 INFO:teuthology.orchestra.run.mira063.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2019-10-01T02:24:36.795836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":6304128,"kb_used_data":12288,"kb_used_omap":0,"kb_used_meta":6291456,"kb_avail":559926912,"statfs":{"total":579820584960,"available":573365157888,"internally_reserved":6442450944,"allocated":12582912,"data_stored":243708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6442450944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884310+0000","last_change":"2019-10-01T02:24:19.883996+0000","last_active":"2019-10-01T02:24:19.884310+0000","last_peered":"2019-10-01T02:24:19.884310+0000","last_clean":"2019-10-01T02:24:19.884310+0000","last_became_active":"2019-10-01T02:24:19.883516+0000","last_became_peered":"2019-10-01T02:24:19.883516+0000","last_unstale":"2019-10-01T02:24:19.884310+0000","last_undegraded":"2019-10-01T02:24:19.884310+0000","last_fullsized":"2019-10-01T02:24:19.884310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881661+0000","last_change":"2019-10-01T02:24:19.880871+0000","last_active":"2019-10-01T02:24:19.881661+0000","last_peered":"2019-10-01T02:24:19.881661+0000","last_clean":"2019-10-01T02:24:19.881661+0000","last_became_active":"2019-10-01T02:24:19.879992+0000","last_became_peered":"2019-10-01T02:24:19.879992+0000","last_unstale":"2019-10-01T02:24:19.881661+0000","last_undegraded":"2019-10-01T02:24:19.881661+0000","last_fullsized":"2019-10-01T02:24:19.881661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884030+0000","last_change":"2019-10-01T02:24:19.883279+0000","last_active":"2019-10-01T02:24:19.884030+0000","last_peered":"2019-10-01T02:24:19.884030+0000","last_clean":"2019-10-01T02:24:19.884030+0000","last_became_active":"2019-10-01T02:24:19.882828+0000","last_became_peered":"2019-10-01T02:24:19.882828+0000","last_unstale":"2019-10-01T02:24:19.884030+0000","last_undegraded":"2019-10-01T02:24:19.884030+0000","last_fullsized":"2019-10-01T02:24:19.884030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.888642+0000","last_change":"2019-10-01T02:24:19.887717+0000","last_active":"2019-10-01T02:24:19.888642+0000","last_peered":"2019-10-01T02:24:19.888642+0000","last_clean":"2019-10-01T02:24:19.888642+0000","last_became_active":"2019-10-01T02:24:19.886121+0000","last_became_peered":"2019-10-01T02:24:19.886121+0000","last_unstale":"2019-10-01T02:24:19.888642+0000","last_undegraded":"2019-10-01T02:24:19.888642+0000","last_fullsized":"2019-10-01T02:24:19.888642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881438+0000","last_change":"2019-10-01T02:24:19.880892+0000","last_active":"2019-10-01T02:24:19.881438+0000","last_peered":"2019-10-01T02:24:19.881438+0000","last_clean":"2019-10-01T02:24:19.881438+0000","last_became_active":"2019-10-01T02:24:19.880369+0000","last_became_peered":"2019-10-01T02:24:19.880369+0000","last_unstale":"2019-10-01T02:24:19.881438+0000","last_undegraded":"2019-10-01T02:24:19.881438+0000","last_fullsized":"2019-10-01T02:24:19.881438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.882972+0000","last_change":"2019-10-01T02:24:19.882549+0000","last_active":"2019-10-01T02:24:19.882972+0000","last_peered":"2019-10-01T02:24:19.882972+0000","last_clean":"2019-10-01T02:24:19.882972+0000","last_became_active":"2019-10-01T02:24:19.882007+0000","last_became_peered":"2019-10-01T02:24:19.882007+0000","last_unstale":"2019-10-01T02:24:19.882972+0000","last_undegraded":"2019-10-01T02:24:19.882972+0000","last_fullsized":"2019-10-01T02:24:19.882972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.883411+0000","last_change":"2019-10-01T02:24:19.883094+0000","last_active":"2019-10-01T02:24:19.883411+0000","last_peered":"2019-10-01T02:24:19.883411+0000","last_clean":"2019-10-01T02:24:19.883411+0000","last_became_active":"2019-10-01T02:24:19.882560+0000","last_became_peered":"2019-10-01T02:24:19.882560+0000","last_unstale":"2019-10-01T02:24:19.883411+0000","last_undegraded":"2019-10-01T02:24:19.883411+0000","last_fullsized":"2019-10-01T02:24:19.883411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.886062+0000","last_change":"2019-10-01T02:24:19.884777+0000","last_active":"2019-10-01T02:24:19.886062+0000","last_peered":"2019-10-01T02:24:19.886062+0000","last_clean":"2019-10-01T02:24:19.886062+0000","last_became_active":"2019-10-01T02:24:19.883514+0000","last_became_peered":"2019-10-01T02:24:19.883514+0000","last_unstale":"2019-10-01T02:24:19.886062+0000","last_undegraded":"2019-10-01T02:24:19.886062+0000","last_fullsized":"2019-10-01T02:24:19.886062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.199}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.933}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.313}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.762}]}]},{"osd":4,"up_from":11,"seq":47244640264,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.262}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.046}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.860}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.117}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.719}]}]},{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.466}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.636}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.929}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.414}]}]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.427}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.348}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.865}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.966}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.884}]}]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.721}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.826}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.155}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.090}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.570}]}]},{"osd":3,"up_from":11,"seq":47244640264,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.982}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.775}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.892}]}]}]}} 2019-10-01T02:24:37.027 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-10-01T02:24:37.027 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-10-01T02:24:37.028 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:37.028 INFO:teuthology.orchestra.run.mira063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2019-10-01T02:24:37.876 INFO:teuthology.misc.health.mira063.stdout:HEALTH_OK 2019-10-01T02:24:37.904 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2019-10-01T02:24:37.904 INFO:teuthology.run_tasks:Running task mon_thrash... 2019-10-01T02:24:37.931 INFO:tasks.mon_thrash:Beginning mon_thrash... 2019-10-01T02:24:37.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:37.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-01T02:24:38.916 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:38.917 INFO:teuthology.orchestra.run.mira063.stdout:{"epoch":15,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","created":"2019-10-01T02:23:57.885259+0000","modified":"2019-10-01T02:24:20.871939+0000","last_up_change":"2019-10-01T02:24:13.019960+0000","last_in_change":"2019-10-01T02:24:13.019960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-01T02:24:18.500383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"eead52b7-7821-4283-b7c2-3ea9c4b1b9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6804","nonce":7027},{"type":"v1","addr":"172.21.6.124:6806","nonce":7027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6810","nonce":7027},{"type":"v1","addr":"172.21.6.124:6813","nonce":7027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6819","nonce":7027},{"type":"v1","addr":"172.21.6.124:6821","nonce":7027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6816","nonce":7027},{"type":"v1","addr":"172.21.6.124:6817","nonce":7027}]},"public_addr":"172.21.6.124:6806/7027","cluster_addr":"172.21.6.124:6813/7027","heartbeat_back_addr":"172.21.6.124:6821/7027","heartbeat_front_addr":"172.21.6.124:6817/7027","state":["exists","up"]},{"osd":1,"uuid":"ab58538f-ffa8-4f9b-85c9-bfbcebc0959d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6800","nonce":7026},{"type":"v1","addr":"172.21.6.124:6801","nonce":7026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6802","nonce":7026},{"type":"v1","addr":"172.21.6.124:6803","nonce":7026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6811","nonce":7026},{"type":"v1","addr":"172.21.6.124:6814","nonce":7026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6805","nonce":7026},{"type":"v1","addr":"172.21.6.124:6808","nonce":7026}]},"public_addr":"172.21.6.124:6801/7026","cluster_addr":"172.21.6.124:6803/7026","heartbeat_back_addr":"172.21.6.124:6814/7026","heartbeat_front_addr":"172.21.6.124:6808/7026","state":["exists","up"]},{"osd":2,"uuid":"243ceb1b-964e-4226-83a5-5bcaeb66e71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6807","nonce":7028},{"type":"v1","addr":"172.21.6.124:6809","nonce":7028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6812","nonce":7028},{"type":"v1","addr":"172.21.6.124:6815","nonce":7028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6822","nonce":7028},{"type":"v1","addr":"172.21.6.124:6823","nonce":7028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:6818","nonce":7028},{"type":"v1","addr":"172.21.6.124:6820","nonce":7028}]},"public_addr":"172.21.6.124:6809/7028","cluster_addr":"172.21.6.124:6815/7028","heartbeat_back_addr":"172.21.6.124:6823/7028","heartbeat_front_addr":"172.21.6.124:6820/7028","state":["exists","up"]},{"osd":3,"uuid":"d13ee4b5-26e5-4b5c-95ed-49770e5613f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6810","nonce":7131},{"type":"v1","addr":"172.21.6.120:6811","nonce":7131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6812","nonce":7131},{"type":"v1","addr":"172.21.6.120:6813","nonce":7131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6819","nonce":7131},{"type":"v1","addr":"172.21.6.120:6821","nonce":7131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6815","nonce":7131},{"type":"v1","addr":"172.21.6.120:6817","nonce":7131}]},"public_addr":"172.21.6.120:6811/7131","cluster_addr":"172.21.6.120:6813/7131","heartbeat_back_addr":"172.21.6.120:6821/7131","heartbeat_front_addr":"172.21.6.120:6817/7131","state":["exists","up"]},{"osd":4,"uuid":"91591ad5-2fc6-49b3-b9dd-4b9ac6019691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6802","nonce":7132},{"type":"v1","addr":"172.21.6.120:6803","nonce":7132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6804","nonce":7132},{"type":"v1","addr":"172.21.6.120:6805","nonce":7132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6808","nonce":7132},{"type":"v1","addr":"172.21.6.120:6809","nonce":7132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6806","nonce":7132},{"type":"v1","addr":"172.21.6.120:6807","nonce":7132}]},"public_addr":"172.21.6.120:6803/7132","cluster_addr":"172.21.6.120:6805/7132","heartbeat_back_addr":"172.21.6.120:6809/7132","heartbeat_front_addr":"172.21.6.120:6807/7132","state":["exists","up"]},{"osd":5,"uuid":"23d48522-d36c-4fe3-8c49-d34479580806","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6814","nonce":7133},{"type":"v1","addr":"172.21.6.120:6816","nonce":7133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6818","nonce":7133},{"type":"v1","addr":"172.21.6.120:6820","nonce":7133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6824","nonce":7133},{"type":"v1","addr":"172.21.6.120:6825","nonce":7133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:6822","nonce":7133},{"type":"v1","addr":"172.21.6.120:6823","nonce":7133}]},"public_addr":"172.21.6.120:6816/7133","cluster_addr":"172.21.6.120:6820/7133","heartbeat_back_addr":"172.21.6.120:6825/7133","heartbeat_front_addr":"172.21.6.120:6823/7133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.194318+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.256827+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.221457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.173318+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.143400+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-01T02:24:11.189943+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-01T02:24:38.938 INFO:tasks.mon_thrash.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-01T02:24:18.500383+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'15', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-01T02:24:38.939 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:38.939 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-01T02:24:39.866 INFO:teuthology.orchestra.run.mira063.stdout:pg_num: 8 2019-10-01T02:24:39.885 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2019-10-01T02:24:39.885 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:39.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-01T02:24:40.827 INFO:teuthology.orchestra.run.mira063.stderr:dumped all 2019-10-01T02:24:40.828 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:40.849 INFO:teuthology.orchestra.run.mira063.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2019-10-01T02:24:40.797112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":6304128,"kb_used_data":12288,"kb_used_omap":0,"kb_used_meta":6291456,"kb_avail":559926912,"statfs":{"total":579820584960,"available":573365157888,"internally_reserved":6442450944,"allocated":12582912,"data_stored":243708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6442450944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884310+0000","last_change":"2019-10-01T02:24:19.883996+0000","last_active":"2019-10-01T02:24:19.884310+0000","last_peered":"2019-10-01T02:24:19.884310+0000","last_clean":"2019-10-01T02:24:19.884310+0000","last_became_active":"2019-10-01T02:24:19.883516+0000","last_became_peered":"2019-10-01T02:24:19.883516+0000","last_unstale":"2019-10-01T02:24:19.884310+0000","last_undegraded":"2019-10-01T02:24:19.884310+0000","last_fullsized":"2019-10-01T02:24:19.884310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881661+0000","last_change":"2019-10-01T02:24:19.880871+0000","last_active":"2019-10-01T02:24:19.881661+0000","last_peered":"2019-10-01T02:24:19.881661+0000","last_clean":"2019-10-01T02:24:19.881661+0000","last_became_active":"2019-10-01T02:24:19.879992+0000","last_became_peered":"2019-10-01T02:24:19.879992+0000","last_unstale":"2019-10-01T02:24:19.881661+0000","last_undegraded":"2019-10-01T02:24:19.881661+0000","last_fullsized":"2019-10-01T02:24:19.881661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884030+0000","last_change":"2019-10-01T02:24:19.883279+0000","last_active":"2019-10-01T02:24:19.884030+0000","last_peered":"2019-10-01T02:24:19.884030+0000","last_clean":"2019-10-01T02:24:19.884030+0000","last_became_active":"2019-10-01T02:24:19.882828+0000","last_became_peered":"2019-10-01T02:24:19.882828+0000","last_unstale":"2019-10-01T02:24:19.884030+0000","last_undegraded":"2019-10-01T02:24:19.884030+0000","last_fullsized":"2019-10-01T02:24:19.884030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.888642+0000","last_change":"2019-10-01T02:24:19.887717+0000","last_active":"2019-10-01T02:24:19.888642+0000","last_peered":"2019-10-01T02:24:19.888642+0000","last_clean":"2019-10-01T02:24:19.888642+0000","last_became_active":"2019-10-01T02:24:19.886121+0000","last_became_peered":"2019-10-01T02:24:19.886121+0000","last_unstale":"2019-10-01T02:24:19.888642+0000","last_undegraded":"2019-10-01T02:24:19.888642+0000","last_fullsized":"2019-10-01T02:24:19.888642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881438+0000","last_change":"2019-10-01T02:24:19.880892+0000","last_active":"2019-10-01T02:24:19.881438+0000","last_peered":"2019-10-01T02:24:19.881438+0000","last_clean":"2019-10-01T02:24:19.881438+0000","last_became_active":"2019-10-01T02:24:19.880369+0000","last_became_peered":"2019-10-01T02:24:19.880369+0000","last_unstale":"2019-10-01T02:24:19.881438+0000","last_undegraded":"2019-10-01T02:24:19.881438+0000","last_fullsized":"2019-10-01T02:24:19.881438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.882972+0000","last_change":"2019-10-01T02:24:19.882549+0000","last_active":"2019-10-01T02:24:19.882972+0000","last_peered":"2019-10-01T02:24:19.882972+0000","last_clean":"2019-10-01T02:24:19.882972+0000","last_became_active":"2019-10-01T02:24:19.882007+0000","last_became_peered":"2019-10-01T02:24:19.882007+0000","last_unstale":"2019-10-01T02:24:19.882972+0000","last_undegraded":"2019-10-01T02:24:19.882972+0000","last_fullsized":"2019-10-01T02:24:19.882972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.883411+0000","last_change":"2019-10-01T02:24:19.883094+0000","last_active":"2019-10-01T02:24:19.883411+0000","last_peered":"2019-10-01T02:24:19.883411+0000","last_clean":"2019-10-01T02:24:19.883411+0000","last_became_active":"2019-10-01T02:24:19.882560+0000","last_became_peered":"2019-10-01T02:24:19.882560+0000","last_unstale":"2019-10-01T02:24:19.883411+0000","last_undegraded":"2019-10-01T02:24:19.883411+0000","last_fullsized":"2019-10-01T02:24:19.883411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.886062+0000","last_change":"2019-10-01T02:24:19.884777+0000","last_active":"2019-10-01T02:24:19.886062+0000","last_peered":"2019-10-01T02:24:19.886062+0000","last_clean":"2019-10-01T02:24:19.886062+0000","last_became_active":"2019-10-01T02:24:19.883514+0000","last_became_peered":"2019-10-01T02:24:19.883514+0000","last_unstale":"2019-10-01T02:24:19.886062+0000","last_undegraded":"2019-10-01T02:24:19.886062+0000","last_fullsized":"2019-10-01T02:24:19.886062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.360}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.836}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.465}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.686}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.886}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.155}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.663}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.795}]}]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.468}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.355}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.161}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.370}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.420}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.160}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.304}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.861}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.516}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.109}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.030}]}]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.099}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.257}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.171}]}]}]}} 2019-10-01T02:24:40.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:40.851 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-01T02:24:41.806 INFO:teuthology.orchestra.run.mira063.stderr:dumped all 2019-10-01T02:24:41.807 INFO:teuthology.orchestra.run.mira063.stdout: 2019-10-01T02:24:41.828 INFO:teuthology.orchestra.run.mira063.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2019-10-01T02:24:40.797112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":6304128,"kb_used_data":12288,"kb_used_omap":0,"kb_used_meta":6291456,"kb_avail":559926912,"statfs":{"total":579820584960,"available":573365157888,"internally_reserved":6442450944,"allocated":12582912,"data_stored":243708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6442450944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884310+0000","last_change":"2019-10-01T02:24:19.883996+0000","last_active":"2019-10-01T02:24:19.884310+0000","last_peered":"2019-10-01T02:24:19.884310+0000","last_clean":"2019-10-01T02:24:19.884310+0000","last_became_active":"2019-10-01T02:24:19.883516+0000","last_became_peered":"2019-10-01T02:24:19.883516+0000","last_unstale":"2019-10-01T02:24:19.884310+0000","last_undegraded":"2019-10-01T02:24:19.884310+0000","last_fullsized":"2019-10-01T02:24:19.884310+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881661+0000","last_change":"2019-10-01T02:24:19.880871+0000","last_active":"2019-10-01T02:24:19.881661+0000","last_peered":"2019-10-01T02:24:19.881661+0000","last_clean":"2019-10-01T02:24:19.881661+0000","last_became_active":"2019-10-01T02:24:19.879992+0000","last_became_peered":"2019-10-01T02:24:19.879992+0000","last_unstale":"2019-10-01T02:24:19.881661+0000","last_undegraded":"2019-10-01T02:24:19.881661+0000","last_fullsized":"2019-10-01T02:24:19.881661+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.884030+0000","last_change":"2019-10-01T02:24:19.883279+0000","last_active":"2019-10-01T02:24:19.884030+0000","last_peered":"2019-10-01T02:24:19.884030+0000","last_clean":"2019-10-01T02:24:19.884030+0000","last_became_active":"2019-10-01T02:24:19.882828+0000","last_became_peered":"2019-10-01T02:24:19.882828+0000","last_unstale":"2019-10-01T02:24:19.884030+0000","last_undegraded":"2019-10-01T02:24:19.884030+0000","last_fullsized":"2019-10-01T02:24:19.884030+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.888642+0000","last_change":"2019-10-01T02:24:19.887717+0000","last_active":"2019-10-01T02:24:19.888642+0000","last_peered":"2019-10-01T02:24:19.888642+0000","last_clean":"2019-10-01T02:24:19.888642+0000","last_became_active":"2019-10-01T02:24:19.886121+0000","last_became_peered":"2019-10-01T02:24:19.886121+0000","last_unstale":"2019-10-01T02:24:19.888642+0000","last_undegraded":"2019-10-01T02:24:19.888642+0000","last_fullsized":"2019-10-01T02:24:19.888642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.881438+0000","last_change":"2019-10-01T02:24:19.880892+0000","last_active":"2019-10-01T02:24:19.881438+0000","last_peered":"2019-10-01T02:24:19.881438+0000","last_clean":"2019-10-01T02:24:19.881438+0000","last_became_active":"2019-10-01T02:24:19.880369+0000","last_became_peered":"2019-10-01T02:24:19.880369+0000","last_unstale":"2019-10-01T02:24:19.881438+0000","last_undegraded":"2019-10-01T02:24:19.881438+0000","last_fullsized":"2019-10-01T02:24:19.881438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.882972+0000","last_change":"2019-10-01T02:24:19.882549+0000","last_active":"2019-10-01T02:24:19.882972+0000","last_peered":"2019-10-01T02:24:19.882972+0000","last_clean":"2019-10-01T02:24:19.882972+0000","last_became_active":"2019-10-01T02:24:19.882007+0000","last_became_peered":"2019-10-01T02:24:19.882007+0000","last_unstale":"2019-10-01T02:24:19.882972+0000","last_undegraded":"2019-10-01T02:24:19.882972+0000","last_fullsized":"2019-10-01T02:24:19.882972+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.883411+0000","last_change":"2019-10-01T02:24:19.883094+0000","last_active":"2019-10-01T02:24:19.883411+0000","last_peered":"2019-10-01T02:24:19.883411+0000","last_clean":"2019-10-01T02:24:19.883411+0000","last_became_active":"2019-10-01T02:24:19.882560+0000","last_became_peered":"2019-10-01T02:24:19.882560+0000","last_unstale":"2019-10-01T02:24:19.883411+0000","last_undegraded":"2019-10-01T02:24:19.883411+0000","last_fullsized":"2019-10-01T02:24:19.883411+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2019-10-01T02:24:19.886062+0000","last_change":"2019-10-01T02:24:19.884777+0000","last_active":"2019-10-01T02:24:19.886062+0000","last_peered":"2019-10-01T02:24:19.886062+0000","last_clean":"2019-10-01T02:24:19.886062+0000","last_became_active":"2019-10-01T02:24:19.883514+0000","last_became_peered":"2019-10-01T02:24:19.883514+0000","last_unstale":"2019-10-01T02:24:19.886062+0000","last_undegraded":"2019-10-01T02:24:19.886062+0000","last_fullsized":"2019-10-01T02:24:19.886062+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-01T02:24:18.855565+0000","last_clean_scrub_stamp":"2019-10-01T02:24:18.855565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.360}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.836}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.465}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.686}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.886}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.219}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.155}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.663}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.795}]}]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.468}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.256}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.355}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.161}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.370}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.420}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.160}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.304}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.861}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.516}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.109}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.030}]}]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050688,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321152,"statfs":{"total":96636764160,"available":95560859648,"internally_reserved":1073741824,"allocated":2097152,"data_stored":40618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.226}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.099}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.257}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.993}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.171}]}]}]}} 2019-10-01T02:24:41.830 INFO:tasks.mon_thrash.ceph_manager:clean! 2019-10-01T02:24:41.831 DEBUG:tasks.mon_thrash:Yielding 2019-10-01T02:24:41.831 INFO:teuthology.run_tasks:Running task rados... 2019-10-01T02:24:41.848 INFO:tasks.rados:Beginning rados... 2019-10-01T02:24:41.848 DEBUG:teuthology.run_tasks:Unwinding manager rados 2019-10-01T02:24:41.883 INFO:tasks.rados:joining rados 2019-10-01T02:24:41.884 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2019-10-01T02:24:41.884 INFO:tasks.mon_thrash.mon_thrasher:seed: 1569896681, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2019-10-01T02:24:41.884 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:24:41.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:41.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:24:41.885 INFO:tasks.rados:clients are ['client.0'] 2019-10-01T02:24:41.885 INFO:tasks.rados:starting run 0 out of 1 2019-10-01T02:24:41.886 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2019-10-01T02:24:41.886 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:41.886 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2019-10-01T02:24:42.836 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":44,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:24:42.865 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":4,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":44,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:24:42.865 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:24:42.865 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:24:42.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:42.865 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:24:42.927 INFO:teuthology.orchestra.run.mira063.stderr:pool 'unique_pool_0' created 2019-10-01T02:24:42.953 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:42.953 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2019-10-01T02:24:43.757 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":4,"quorum":[0,1,2],"quorum_age":45,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:24:43.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:43.779 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:24:43.908 INFO:teuthology.orchestra.run.mira063.stderr:enabled application 'rados' on pool 'unique_pool_0' 2019-10-01T02:24:44.707 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":4,"quorum":[0,1,2],"quorum_age":46,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:24:44.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:44.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:24:44.932 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:24:44.933 INFO:teuthology.orchestra.run.mira064:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op read 100 --op copy_from 50 --op write 50 --op write_excl 50 --op delete 50 --pool unique_pool_0 2019-10-01T02:24:45.079 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight snap_remove -> 50 2019-10-01T02:24:45.079 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight snap_create -> 50 2019-10-01T02:24:45.079 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight rollback -> 50 2019-10-01T02:24:45.079 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight read -> 100 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight copy_from -> 50 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight write -> 50 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight write_excl -> 50 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:adding op weight delete -> 50 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout:Configuration: 2019-10-01T02:24:45.080 INFO:tasks.rados.rados.0.mira064.stdout: Number of operations: 4000 2019-10-01T02:24:45.081 INFO:tasks.rados.rados.0.mira064.stdout: Number of objects: 50 2019-10-01T02:24:45.081 INFO:tasks.rados.rados.0.mira064.stdout: Max in flight operations: 16 2019-10-01T02:24:45.081 INFO:tasks.rados.rados.0.mira064.stdout: Object size (in bytes): 4000000 2019-10-01T02:24:45.081 INFO:tasks.rados.rados.0.mira064.stdout: Write stride min: 400000 2019-10-01T02:24:45.081 INFO:tasks.rados.rados.0.mira064.stdout: Write stride max: 800000 2019-10-01T02:24:45.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":4,"quorum":[0,1,2],"quorum_age":47,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:24:45.671 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:24:45.671 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:24:45.671 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:24:45.671 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:24:45.672 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:24:45.672 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:24:45.692 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:24:45.692 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:24:45.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:45.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:24:46.662 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":48,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:24:46.692 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":4,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":48,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:24:49.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:24:49.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:25:00.029 INFO:tasks.rados.rados.0.mira064.stdout:1: write initial oid 1 2019-10-01T02:25:00.030 INFO:tasks.rados.rados.0.mira064.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2019-10-01T02:25:00.048 INFO:tasks.rados.rados.0.mira064.stdout:1: writing mira0647520-1 from 416807 to 892056 tid 1 2019-10-01T02:25:00.070 INFO:tasks.rados.rados.0.mira064.stdout:1: writing mira0647520-1 from 1542129 to 2016807 tid 2 2019-10-01T02:25:00.070 INFO:tasks.rados.rados.0.mira064.stdout:2: write initial oid 2 2019-10-01T02:25:00.070 INFO:tasks.rados.rados.0.mira064.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2019-10-01T02:25:00.086 INFO:tasks.rados.rados.0.mira064.stdout:2: writing mira0647520-2 from 416807 to 892056 tid 1 2019-10-01T02:25:00.105 INFO:tasks.rados.rados.0.mira064.stdout:2: writing mira0647520-2 from 1542129 to 2016807 tid 2 2019-10-01T02:25:00.105 INFO:tasks.rados.rados.0.mira064.stdout:3: write initial oid 3 2019-10-01T02:25:00.105 INFO:tasks.rados.rados.0.mira064.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2019-10-01T02:25:00.121 INFO:tasks.rados.rados.0.mira064.stdout:3: writing mira0647520-3 from 433614 to 984112 tid 1 2019-10-01T02:25:00.136 INFO:tasks.rados.rados.0.mira064.stdout:3: writing mira0647520-3 from 1600611 to 2033614 tid 2 2019-10-01T02:25:00.136 INFO:tasks.rados.rados.0.mira064.stdout:4: write initial oid 4 2019-10-01T02:25:00.137 INFO:tasks.rados.rados.0.mira064.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2019-10-01T02:25:00.151 INFO:tasks.rados.rados.0.mira064.stdout:4: writing mira0647520-4 from 450421 to 1076168 tid 1 2019-10-01T02:25:00.164 INFO:tasks.rados.rados.0.mira064.stdout:4: writing mira0647520-4 from 1659093 to 2050421 tid 2 2019-10-01T02:25:00.164 INFO:tasks.rados.rados.0.mira064.stdout:5: write initial oid 5 2019-10-01T02:25:00.165 INFO:tasks.rados.rados.0.mira064.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2019-10-01T02:25:00.179 INFO:tasks.rados.rados.0.mira064.stdout:5: writing mira0647520-5 from 467228 to 1168224 tid 1 2019-10-01T02:25:00.190 INFO:tasks.rados.rados.0.mira064.stdout:5: writing mira0647520-5 from 1717575 to 2067228 tid 2 2019-10-01T02:25:00.190 INFO:tasks.rados.rados.0.mira064.stdout:6: write initial oid 6 2019-10-01T02:25:00.190 INFO:tasks.rados.rados.0.mira064.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2019-10-01T02:25:00.205 INFO:tasks.rados.rados.0.mira064.stdout:6: writing mira0647520-6 from 484035 to 1260280 tid 1 2019-10-01T02:25:00.215 INFO:tasks.rados.rados.0.mira064.stdout:6: writing mira0647520-6 from 2059704 to 2084035 tid 2 2019-10-01T02:25:00.215 INFO:tasks.rados.rados.0.mira064.stdout:7: write initial oid 7 2019-10-01T02:25:00.215 INFO:tasks.rados.rados.0.mira064.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2019-10-01T02:25:00.225 INFO:tasks.rados.rados.0.mira064.stdout:7: writing mira0647520-7 from 500842 to 952336 tid 1 2019-10-01T02:25:00.241 INFO:tasks.rados.rados.0.mira064.stdout:7: writing mira0647520-7 from 1718186 to 2100842 tid 2 2019-10-01T02:25:00.241 INFO:tasks.rados.rados.0.mira064.stdout:8: write initial oid 8 2019-10-01T02:25:00.241 INFO:tasks.rados.rados.0.mira064.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2019-10-01T02:25:00.248 INFO:tasks.rados.rados.0.mira064.stdout:8: writing mira0647520-8 from 517649 to 1044392 tid 1 2019-10-01T02:25:00.257 INFO:tasks.rados.rados.0.mira064.stdout:8: writing mira0647520-8 from 1776668 to 2117649 tid 2 2019-10-01T02:25:00.257 INFO:tasks.rados.rados.0.mira064.stdout:9: write initial oid 9 2019-10-01T02:25:00.258 INFO:tasks.rados.rados.0.mira064.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2019-10-01T02:25:00.269 INFO:tasks.rados.rados.0.mira064.stdout:9: writing mira0647520-9 from 534456 to 1252801 tid 1 2019-10-01T02:25:00.282 INFO:tasks.rados.rados.0.mira064.stdout:9: writing mira0647520-9 from 1951503 to 2134456 tid 2 2019-10-01T02:25:00.282 INFO:tasks.rados.rados.0.mira064.stdout:10: write initial oid 10 2019-10-01T02:25:00.283 INFO:tasks.rados.rados.0.mira064.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2019-10-01T02:25:00.290 INFO:tasks.rados.rados.0.mira064.stdout:10: writing mira0647520-10 from 551263 to 1344857 tid 1 2019-10-01T02:25:00.298 INFO:tasks.rados.rados.0.mira064.stdout:10: writing mira0647520-10 from 1893632 to 2151263 tid 2 2019-10-01T02:25:00.298 INFO:tasks.rados.rados.0.mira064.stdout:11: write initial oid 11 2019-10-01T02:25:00.299 INFO:tasks.rados.rados.0.mira064.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2019-10-01T02:25:00.309 INFO:tasks.rados.rados.0.mira064.stdout:11: writing mira0647520-11 from 568070 to 1036913 tid 1 2019-10-01T02:25:00.320 INFO:tasks.rados.rados.0.mira064.stdout:11: writing mira0647520-11 from 1552114 to 2168070 tid 2 2019-10-01T02:25:00.320 INFO:tasks.rados.rados.0.mira064.stdout:12: write initial oid 12 2019-10-01T02:25:00.320 INFO:tasks.rados.rados.0.mira064.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2019-10-01T02:25:00.330 INFO:tasks.rados.rados.0.mira064.stdout:12: writing mira0647520-12 from 584877 to 1128969 tid 1 2019-10-01T02:25:00.341 INFO:tasks.rados.rados.0.mira064.stdout:12: writing mira0647520-12 from 1610596 to 2172599 tid 2 2019-10-01T02:25:00.341 INFO:tasks.rados.rados.0.mira064.stdout:13: write initial oid 13 2019-10-01T02:25:00.341 INFO:tasks.rados.rados.0.mira064.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2019-10-01T02:25:00.353 INFO:tasks.rados.rados.0.mira064.stdout:13: writing mira0647520-13 from 601684 to 1221025 tid 1 2019-10-01T02:25:00.364 INFO:tasks.rados.rados.0.mira064.stdout:13: writing mira0647520-13 from 1669078 to 2201684 tid 2 2019-10-01T02:25:00.365 INFO:tasks.rados.rados.0.mira064.stdout:14: write initial oid 14 2019-10-01T02:25:00.365 INFO:tasks.rados.rados.0.mira064.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2019-10-01T02:25:00.377 INFO:tasks.rados.rados.0.mira064.stdout:14: writing mira0647520-14 from 618491 to 1313081 tid 1 2019-10-01T02:25:00.387 INFO:tasks.rados.rados.0.mira064.stdout:14: writing mira0647520-14 from 2011207 to 2218491 tid 2 2019-10-01T02:25:00.387 INFO:tasks.rados.rados.0.mira064.stdout:15: write initial oid 15 2019-10-01T02:25:00.387 INFO:tasks.rados.rados.0.mira064.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2019-10-01T02:25:00.401 INFO:tasks.rados.rados.0.mira064.stdout:15: writing mira0647520-15 from 635298 to 1405137 tid 1 2019-10-01T02:25:00.409 INFO:tasks.rados.rados.0.mira064.stdout:15: writing mira0647520-15 from 2069689 to 2235298 tid 2 2019-10-01T02:25:00.409 INFO:tasks.rados.rados.0.mira064.stdout:16: write initial oid 16 2019-10-01T02:25:00.410 INFO:tasks.rados.rados.0.mira064.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2019-10-01T02:25:00.420 INFO:tasks.rados.rados.0.mira064.stdout:16: writing mira0647520-16 from 652105 to 1097193 tid 1 2019-10-01T02:25:00.430 INFO:tasks.rados.rados.0.mira064.stdout:16: writing mira0647520-16 from 1728171 to 2181159 tid 2 2019-10-01T02:25:00.430 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.432 INFO:tasks.rados.rados.0.mira064.stdout:1: finishing write tid 1 to mira0647520-1 2019-10-01T02:25:00.432 INFO:tasks.rados.rados.0.mira064.stdout:1: finishing write tid 2 to mira0647520-1 2019-10-01T02:25:00.432 INFO:tasks.rados.rados.0.mira064.stdout:1: finishing write tid 4 to mira0647520-1 2019-10-01T02:25:00.433 INFO:tasks.rados.rados.0.mira064.stdout:1: finishing write tid 5 to mira0647520-1 2019-10-01T02:25:00.433 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2019-10-01T02:25:00.433 INFO:tasks.rados.rados.0.mira064.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2019-10-01T02:25:00.433 INFO:tasks.rados.rados.0.mira064.stdout:2: finishing write tid 1 to mira0647520-2 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:2: finishing write tid 2 to mira0647520-2 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:2: finishing write tid 4 to mira0647520-2 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:2: finishing write tid 5 to mira0647520-2 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 2 v 6 (ObjNum 1 snap 0 seq_num 1) dirty exists 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:3: finishing write tid 1 to mira0647520-3 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:3: finishing write tid 2 to mira0647520-3 2019-10-01T02:25:00.434 INFO:tasks.rados.rados.0.mira064.stdout:3: finishing write tid 4 to mira0647520-3 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:3: finishing write tid 5 to mira0647520-3 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 3 v 9 (ObjNum 2 snap 0 seq_num 2) dirty exists 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:4: finishing write tid 1 to mira0647520-4 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:4: finishing write tid 2 to mira0647520-4 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:4: finishing write tid 4 to mira0647520-4 2019-10-01T02:25:00.435 INFO:tasks.rados.rados.0.mira064.stdout:1: done (15 left) 2019-10-01T02:25:00.436 INFO:tasks.rados.rados.0.mira064.stdout:2: done (14 left) 2019-10-01T02:25:00.436 INFO:tasks.rados.rados.0.mira064.stdout:3: done (13 left) 2019-10-01T02:25:00.436 INFO:tasks.rados.rados.0.mira064.stdout:17: write initial oid 17 2019-10-01T02:25:00.436 INFO:tasks.rados.rados.0.mira064.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2019-10-01T02:25:00.454 INFO:tasks.rados.rados.0.mira064.stdout:17: writing mira0647520-17 from 668912 to 1305602 tid 1 2019-10-01T02:25:00.455 INFO:tasks.rados.rados.0.mira064.stdout:17: writing mira0647520-17 from 1903006 to 2268912 tid 2 2019-10-01T02:25:00.455 INFO:tasks.rados.rados.0.mira064.stdout:18: write initial oid 18 2019-10-01T02:25:00.455 INFO:tasks.rados.rados.0.mira064.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2019-10-01T02:25:00.467 INFO:tasks.rados.rados.0.mira064.stdout:18: writing mira0647520-18 from 685719 to 1397658 tid 1 2019-10-01T02:25:00.476 INFO:tasks.rados.rados.0.mira064.stdout:18: writing mira0647520-18 from 1845135 to 2285719 tid 2 2019-10-01T02:25:00.476 INFO:tasks.rados.rados.0.mira064.stdout:19: write initial oid 19 2019-10-01T02:25:00.477 INFO:tasks.rados.rados.0.mira064.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2019-10-01T02:25:00.489 INFO:tasks.rados.rados.0.mira064.stdout:19: writing mira0647520-19 from 702526 to 1489714 tid 1 2019-10-01T02:25:00.497 INFO:tasks.rados.rados.0.mira064.stdout:19: writing mira0647520-19 from 1903617 to 2302526 tid 2 2019-10-01T02:25:00.497 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.497 INFO:tasks.rados.rados.0.mira064.stdout:4: finishing write tid 5 to mira0647520-4 2019-10-01T02:25:00.498 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2019-10-01T02:25:00.498 INFO:tasks.rados.rados.0.mira064.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2019-10-01T02:25:00.499 INFO:tasks.rados.rados.0.mira064.stdout:4: done (15 left) 2019-10-01T02:25:00.499 INFO:tasks.rados.rados.0.mira064.stdout:20: write initial oid 20 2019-10-01T02:25:00.499 INFO:tasks.rados.rados.0.mira064.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2019-10-01T02:25:00.508 INFO:tasks.rados.rados.0.mira064.stdout:20: writing mira0647520-20 from 719333 to 1181770 tid 1 2019-10-01T02:25:00.521 INFO:tasks.rados.rados.0.mira064.stdout:20: writing mira0647520-20 from 1962099 to 2319333 tid 2 2019-10-01T02:25:00.521 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.521 INFO:tasks.rados.rados.0.mira064.stdout:5: finishing write tid 1 to mira0647520-5 2019-10-01T02:25:00.522 INFO:tasks.rados.rados.0.mira064.stdout:5: finishing write tid 2 to mira0647520-5 2019-10-01T02:25:00.522 INFO:tasks.rados.rados.0.mira064.stdout:5: finishing write tid 4 to mira0647520-5 2019-10-01T02:25:00.522 INFO:tasks.rados.rados.0.mira064.stdout:5: finishing write tid 5 to mira0647520-5 2019-10-01T02:25:00.522 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 5 v 12 (ObjNum 4 snap 0 seq_num 4) dirty exists 2019-10-01T02:25:00.523 INFO:tasks.rados.rados.0.mira064.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2019-10-01T02:25:00.523 INFO:tasks.rados.rados.0.mira064.stdout:6: finishing write tid 1 to mira0647520-6 2019-10-01T02:25:00.523 INFO:tasks.rados.rados.0.mira064.stdout:5: done (15 left) 2019-10-01T02:25:00.523 INFO:tasks.rados.rados.0.mira064.stdout:21: write initial oid 21 2019-10-01T02:25:00.523 INFO:tasks.rados.rados.0.mira064.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2019-10-01T02:25:00.534 INFO:tasks.rados.rados.0.mira064.stdout:21: writing mira0647520-21 from 736140 to 1273826 tid 1 2019-10-01T02:25:00.544 INFO:tasks.rados.rados.0.mira064.stdout:21: writing mira0647520-21 from 2020581 to 2336140 tid 2 2019-10-01T02:25:00.545 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.546 INFO:tasks.rados.rados.0.mira064.stdout:6: finishing write tid 2 to mira0647520-6 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:6: finishing write tid 4 to mira0647520-6 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:6: finishing write tid 5 to mira0647520-6 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:7: finishing write tid 1 to mira0647520-7 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:7: finishing write tid 2 to mira0647520-7 2019-10-01T02:25:00.547 INFO:tasks.rados.rados.0.mira064.stdout:7: finishing write tid 4 to mira0647520-7 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:7: finishing write tid 5 to mira0647520-7 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:8: finishing write tid 1 to mira0647520-8 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:8: finishing write tid 2 to mira0647520-8 2019-10-01T02:25:00.548 INFO:tasks.rados.rados.0.mira064.stdout:8: finishing write tid 4 to mira0647520-8 2019-10-01T02:25:00.549 INFO:tasks.rados.rados.0.mira064.stdout:6: done (15 left) 2019-10-01T02:25:00.549 INFO:tasks.rados.rados.0.mira064.stdout:7: done (14 left) 2019-10-01T02:25:00.549 INFO:tasks.rados.rados.0.mira064.stdout:22: write initial oid 22 2019-10-01T02:25:00.549 INFO:tasks.rados.rados.0.mira064.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2019-10-01T02:25:00.558 INFO:tasks.rados.rados.0.mira064.stdout:22: writing mira0647520-22 from 752947 to 1365882 tid 1 2019-10-01T02:25:00.568 INFO:tasks.rados.rados.0.mira064.stdout:22: writing mira0647520-22 from 1962710 to 2352947 tid 2 2019-10-01T02:25:00.568 INFO:tasks.rados.rados.0.mira064.stdout:23: write initial oid 23 2019-10-01T02:25:00.569 INFO:tasks.rados.rados.0.mira064.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2019-10-01T02:25:00.583 INFO:tasks.rados.rados.0.mira064.stdout:23: writing mira0647520-23 from 769754 to 1457938 tid 1 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:23: writing mira0647520-23 from 2021192 to 2369754 tid 2 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:8: finishing write tid 5 to mira0647520-8 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 8 v 15 (ObjNum 7 snap 0 seq_num 7) dirty exists 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:9: finishing write tid 1 to mira0647520-9 2019-10-01T02:25:00.593 INFO:tasks.rados.rados.0.mira064.stdout:9: finishing write tid 2 to mira0647520-9 2019-10-01T02:25:00.594 INFO:tasks.rados.rados.0.mira064.stdout:9: finishing write tid 4 to mira0647520-9 2019-10-01T02:25:00.594 INFO:tasks.rados.rados.0.mira064.stdout:8: done (15 left) 2019-10-01T02:25:00.594 INFO:tasks.rados.rados.0.mira064.stdout:24: write initial oid 24 2019-10-01T02:25:00.594 INFO:tasks.rados.rados.0.mira064.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2019-10-01T02:25:00.605 INFO:tasks.rados.rados.0.mira064.stdout:24: writing mira0647520-24 from 786561 to 1266347 tid 1 2019-10-01T02:25:00.610 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:25:00.615 INFO:tasks.rados.rados.0.mira064.stdout:24: writing mira0647520-24 from 1796027 to 2263691 tid 2 2019-10-01T02:25:00.616 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.616 INFO:tasks.rados.rados.0.mira064.stdout:9: finishing write tid 5 to mira0647520-9 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:10: finishing write tid 1 to mira0647520-10 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:10: finishing write tid 2 to mira0647520-10 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:10: finishing write tid 4 to mira0647520-10 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:10: finishing write tid 5 to mira0647520-10 2019-10-01T02:25:00.617 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:11: finishing write tid 1 to mira0647520-11 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:11: finishing write tid 2 to mira0647520-11 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:9: done (15 left) 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:10: done (14 left) 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:25: write initial oid 25 2019-10-01T02:25:00.618 INFO:tasks.rados.rados.0.mira064.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2019-10-01T02:25:00.625 INFO:tasks.rados.rados.0.mira064.stdout:25: writing mira0647520-25 from 403368 to 958403 tid 1 2019-10-01T02:25:00.629 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":6,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:25:00.630 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:25:00.630 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:25:00.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:25:00.638 INFO:tasks.rados.rados.0.mira064.stdout:25: writing mira0647520-25 from 1454509 to 2182184 tid 2 2019-10-01T02:25:00.638 INFO:tasks.rados.rados.0.mira064.stdout:26: write initial oid 26 2019-10-01T02:25:00.638 INFO:tasks.rados.rados.0.mira064.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2019-10-01T02:25:00.647 INFO:tasks.rados.rados.0.mira064.stdout:26: writing mira0647520-26 from 420175 to 1050459 tid 1 2019-10-01T02:25:00.660 INFO:tasks.rados.rados.0.mira064.stdout:26: writing mira0647520-26 from 1796638 to 2267971 tid 2 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:11: finishing write tid 4 to mira0647520-11 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:11: finishing write tid 5 to mira0647520-11 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:12: finishing write tid 1 to mira0647520-12 2019-10-01T02:25:00.661 INFO:tasks.rados.rados.0.mira064.stdout:12: finishing write tid 2 to mira0647520-12 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:12: finishing write tid 4 to mira0647520-12 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:12: finishing write tid 5 to mira0647520-12 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 12 v 18 (ObjNum 11 snap 0 seq_num 11) dirty exists 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:13: finishing write tid 1 to mira0647520-13 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:13: finishing write tid 2 to mira0647520-13 2019-10-01T02:25:00.662 INFO:tasks.rados.rados.0.mira064.stdout:11: done (15 left) 2019-10-01T02:25:00.663 INFO:tasks.rados.rados.0.mira064.stdout:12: done (14 left) 2019-10-01T02:25:00.663 INFO:tasks.rados.rados.0.mira064.stdout:27: write initial oid 27 2019-10-01T02:25:00.663 INFO:tasks.rados.rados.0.mira064.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2019-10-01T02:25:00.671 INFO:tasks.rados.rados.0.mira064.stdout:27: writing mira0647520-27 from 436982 to 1142515 tid 1 2019-10-01T02:25:00.684 INFO:tasks.rados.rados.0.mira064.stdout:27: writing mira0647520-27 from 1855120 to 2436982 tid 2 2019-10-01T02:25:00.684 INFO:tasks.rados.rados.0.mira064.stdout:28: write initial oid 28 2019-10-01T02:25:00.684 INFO:tasks.rados.rados.0.mira064.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2019-10-01T02:25:00.696 INFO:tasks.rados.rados.0.mira064.stdout:28: writing mira0647520-28 from 453789 to 1234571 tid 1 2019-10-01T02:25:00.707 INFO:tasks.rados.rados.0.mira064.stdout:28: writing mira0647520-28 from 1913602 to 2388604 tid 2 2019-10-01T02:25:00.708 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.708 INFO:tasks.rados.rados.0.mira064.stdout:13: finishing write tid 4 to mira0647520-13 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:13: finishing write tid 5 to mira0647520-13 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:14: finishing write tid 1 to mira0647520-14 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:14: finishing write tid 2 to mira0647520-14 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:14: finishing write tid 4 to mira0647520-14 2019-10-01T02:25:00.709 INFO:tasks.rados.rados.0.mira064.stdout:14: finishing write tid 5 to mira0647520-14 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:15: finishing write tid 1 to mira0647520-15 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:15: finishing write tid 2 to mira0647520-15 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:15: finishing write tid 4 to mira0647520-15 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:15: finishing write tid 5 to mira0647520-15 2019-10-01T02:25:00.710 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:16: finishing write tid 1 to mira0647520-16 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:13: done (15 left) 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:14: done (14 left) 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:15: done (13 left) 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:29: write initial oid 29 2019-10-01T02:25:00.711 INFO:tasks.rados.rados.0.mira064.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2019-10-01T02:25:00.717 INFO:tasks.rados.rados.0.mira064.stdout:29: writing mira0647520-29 from 470596 to 926627 tid 1 2019-10-01T02:25:00.729 INFO:tasks.rados.rados.0.mira064.stdout:29: writing mira0647520-29 from 1572084 to 2190744 tid 2 2019-10-01T02:25:00.730 INFO:tasks.rados.rados.0.mira064.stdout:30: write initial oid 30 2019-10-01T02:25:00.730 INFO:tasks.rados.rados.0.mira064.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2019-10-01T02:25:00.738 INFO:tasks.rados.rados.0.mira064.stdout:30: writing mira0647520-30 from 487403 to 1018683 tid 1 2019-10-01T02:25:00.752 INFO:tasks.rados.rados.0.mira064.stdout:30: writing mira0647520-30 from 1514213 to 1992884 tid 2 2019-10-01T02:25:00.753 INFO:tasks.rados.rados.0.mira064.stdout:31: write initial oid 31 2019-10-01T02:25:00.753 INFO:tasks.rados.rados.0.mira064.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2019-10-01T02:25:00.761 INFO:tasks.rados.rados.0.mira064.stdout:31: writing mira0647520-31 from 504210 to 1110739 tid 1 2019-10-01T02:25:00.770 INFO:tasks.rados.rados.0.mira064.stdout:31: writing mira0647520-31 from 1572695 to 2195024 tid 2 2019-10-01T02:25:00.770 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.773 INFO:tasks.rados.rados.0.mira064.stdout:16: finishing write tid 2 to mira0647520-16 2019-10-01T02:25:00.773 INFO:tasks.rados.rados.0.mira064.stdout:16: finishing write tid 4 to mira0647520-16 2019-10-01T02:25:00.773 INFO:tasks.rados.rados.0.mira064.stdout:16: finishing write tid 5 to mira0647520-16 2019-10-01T02:25:00.773 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2019-10-01T02:25:00.773 INFO:tasks.rados.rados.0.mira064.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:17: finishing write tid 1 to mira0647520-17 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:17: finishing write tid 2 to mira0647520-17 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:17: finishing write tid 4 to mira0647520-17 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:17: finishing write tid 5 to mira0647520-17 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2019-10-01T02:25:00.774 INFO:tasks.rados.rados.0.mira064.stdout:18: finishing write tid 1 to mira0647520-18 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:18: finishing write tid 2 to mira0647520-18 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:18: finishing write tid 4 to mira0647520-18 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:16: done (15 left) 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:17: done (14 left) 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:32: write initial oid 32 2019-10-01T02:25:00.775 INFO:tasks.rados.rados.0.mira064.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2019-10-01T02:25:00.782 INFO:tasks.rados.rados.0.mira064.stdout:32: writing mira0647520-32 from 521017 to 1319148 tid 1 2019-10-01T02:25:00.800 INFO:tasks.rados.rados.0.mira064.stdout:32: writing mira0647520-32 from 1747530 to 2229870 tid 2 2019-10-01T02:25:00.800 INFO:tasks.rados.rados.0.mira064.stdout:33: write initial oid 33 2019-10-01T02:25:00.801 INFO:tasks.rados.rados.0.mira064.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2019-10-01T02:25:00.819 INFO:tasks.rados.rados.0.mira064.stdout:33: writing mira0647520-33 from 537824 to 1011204 tid 1 2019-10-01T02:25:00.846 INFO:tasks.rados.rados.0.mira064.stdout:33: writing mira0647520-33 from 1806012 to 2432010 tid 2 2019-10-01T02:25:00.847 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.848 INFO:tasks.rados.rados.0.mira064.stdout:18: finishing write tid 5 to mira0647520-18 2019-10-01T02:25:00.848 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:19: finishing write tid 1 to mira0647520-19 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:19: finishing write tid 2 to mira0647520-19 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:19: finishing write tid 4 to mira0647520-19 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:19: finishing write tid 5 to mira0647520-19 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2019-10-01T02:25:00.849 INFO:tasks.rados.rados.0.mira064.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:20: finishing write tid 1 to mira0647520-20 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:21: finishing write tid 1 to mira0647520-21 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:20: finishing write tid 2 to mira0647520-20 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:18: done (15 left) 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:19: done (14 left) 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:34: write initial oid 34 2019-10-01T02:25:00.850 INFO:tasks.rados.rados.0.mira064.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2019-10-01T02:25:00.866 INFO:tasks.rados.rados.0.mira064.stdout:34: writing mira0647520-34 from 554631 to 1103260 tid 1 2019-10-01T02:25:00.884 INFO:tasks.rados.rados.0.mira064.stdout:34: writing mira0647520-34 from 1748141 to 2234150 tid 2 2019-10-01T02:25:00.884 INFO:tasks.rados.rados.0.mira064.stdout:35: write initial oid 35 2019-10-01T02:25:00.884 INFO:tasks.rados.rados.0.mira064.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2019-10-01T02:25:00.902 INFO:tasks.rados.rados.0.mira064.stdout:35: writing mira0647520-35 from 571438 to 1195316 tid 1 2019-10-01T02:25:00.920 INFO:tasks.rados.rados.0.mira064.stdout:35: writing mira0647520-35 from 1806623 to 2436290 tid 2 2019-10-01T02:25:00.920 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.920 INFO:tasks.rados.rados.0.mira064.stdout:20: finishing write tid 4 to mira0647520-20 2019-10-01T02:25:00.920 INFO:tasks.rados.rados.0.mira064.stdout:20: finishing write tid 5 to mira0647520-20 2019-10-01T02:25:00.920 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2019-10-01T02:25:00.921 INFO:tasks.rados.rados.0.mira064.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2019-10-01T02:25:00.921 INFO:tasks.rados.rados.0.mira064.stdout:21: finishing write tid 2 to mira0647520-21 2019-10-01T02:25:00.921 INFO:tasks.rados.rados.0.mira064.stdout:21: finishing write tid 4 to mira0647520-21 2019-10-01T02:25:00.921 INFO:tasks.rados.rados.0.mira064.stdout:21: finishing write tid 5 to mira0647520-21 2019-10-01T02:25:00.921 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 21 v 21 (ObjNum 20 snap 0 seq_num 20) dirty exists 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:22: finishing write tid 1 to mira0647520-22 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:22: finishing write tid 2 to mira0647520-22 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:22: finishing write tid 4 to mira0647520-22 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:20: done (15 left) 2019-10-01T02:25:00.922 INFO:tasks.rados.rados.0.mira064.stdout:21: done (14 left) 2019-10-01T02:25:00.923 INFO:tasks.rados.rados.0.mira064.stdout:36: write initial oid 36 2019-10-01T02:25:00.923 INFO:tasks.rados.rados.0.mira064.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2019-10-01T02:25:00.935 INFO:tasks.rados.rados.0.mira064.stdout:36: writing mira0647520-36 from 588245 to 1287372 tid 1 2019-10-01T02:25:00.947 INFO:tasks.rados.rados.0.mira064.stdout:36: writing mira0647520-36 from 1865105 to 2354783 tid 2 2019-10-01T02:25:00.947 INFO:tasks.rados.rados.0.mira064.stdout:37: write initial oid 37 2019-10-01T02:25:00.948 INFO:tasks.rados.rados.0.mira064.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2019-10-01T02:25:00.964 INFO:tasks.rados.rados.0.mira064.stdout:37: writing mira0647520-37 from 605052 to 1379428 tid 1 2019-10-01T02:25:00.978 INFO:tasks.rados.rados.0.mira064.stdout:37: writing mira0647520-37 from 1923587 to 2556923 tid 2 2019-10-01T02:25:00.979 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:00.980 INFO:tasks.rados.rados.0.mira064.stdout:22: finishing write tid 5 to mira0647520-22 2019-10-01T02:25:00.980 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2019-10-01T02:25:00.980 INFO:tasks.rados.rados.0.mira064.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:23: finishing write tid 1 to mira0647520-23 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:23: finishing write tid 2 to mira0647520-23 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:23: finishing write tid 4 to mira0647520-23 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:23: finishing write tid 5 to mira0647520-23 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2019-10-01T02:25:00.981 INFO:tasks.rados.rados.0.mira064.stdout:24: finishing write tid 1 to mira0647520-24 2019-10-01T02:25:00.982 INFO:tasks.rados.rados.0.mira064.stdout:24: finishing write tid 2 to mira0647520-24 2019-10-01T02:25:00.982 INFO:tasks.rados.rados.0.mira064.stdout:24: finishing write tid 4 to mira0647520-24 2019-10-01T02:25:00.982 INFO:tasks.rados.rados.0.mira064.stdout:24: finishing write tid 5 to mira0647520-24 2019-10-01T02:25:00.982 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2019-10-01T02:25:00.982 INFO:tasks.rados.rados.0.mira064.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:25: finishing write tid 1 to mira0647520-25 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:26: finishing write tid 1 to mira0647520-26 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:25: finishing write tid 2 to mira0647520-25 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:25: finishing write tid 4 to mira0647520-25 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:25: finishing write tid 5 to mira0647520-25 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2019-10-01T02:25:00.983 INFO:tasks.rados.rados.0.mira064.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:27: finishing write tid 1 to mira0647520-27 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:22: done (15 left) 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:23: done (14 left) 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:24: done (13 left) 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:25: done (12 left) 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:38: write initial oid 38 2019-10-01T02:25:00.984 INFO:tasks.rados.rados.0.mira064.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2019-10-01T02:25:00.990 INFO:tasks.rados.rados.0.mira064.stdout:38: writing mira0647520-38 from 621859 to 1071484 tid 1 2019-10-01T02:25:01.006 INFO:tasks.rados.rados.0.mira064.stdout:38: writing mira0647520-38 from 1865716 to 2621859 tid 2 2019-10-01T02:25:01.006 INFO:tasks.rados.rados.0.mira064.stdout:39: write initial oid 39 2019-10-01T02:25:01.006 INFO:tasks.rados.rados.0.mira064.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2019-10-01T02:25:01.017 INFO:tasks.rados.rados.0.mira064.stdout:39: writing mira0647520-39 from 638666 to 1163540 tid 1 2019-10-01T02:25:01.031 INFO:tasks.rados.rados.0.mira064.stdout:39: writing mira0647520-39 from 1924198 to 2561203 tid 2 2019-10-01T02:25:01.031 INFO:tasks.rados.rados.0.mira064.stdout:40: write initial oid 40 2019-10-01T02:25:01.031 INFO:tasks.rados.rados.0.mira064.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2019-10-01T02:25:01.044 INFO:tasks.rados.rados.0.mira064.stdout:40: writing mira0647520-40 from 655473 to 1371949 tid 1 2019-10-01T02:25:01.057 INFO:tasks.rados.rados.0.mira064.stdout:40: writing mira0647520-40 from 2099033 to 2655473 tid 2 2019-10-01T02:25:01.057 INFO:tasks.rados.rados.0.mira064.stdout:41: write initial oid 41 2019-10-01T02:25:01.057 INFO:tasks.rados.rados.0.mira064.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2019-10-01T02:25:01.071 INFO:tasks.rados.rados.0.mira064.stdout:41: writing mira0647520-41 from 672280 to 1464005 tid 1 2019-10-01T02:25:01.082 INFO:tasks.rados.rados.0.mira064.stdout:41: writing mira0647520-41 from 2157515 to 2672280 tid 2 2019-10-01T02:25:01.083 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.083 INFO:tasks.rados.rados.0.mira064.stdout:26: finishing write tid 2 to mira0647520-26 2019-10-01T02:25:01.083 INFO:tasks.rados.rados.0.mira064.stdout:26: finishing write tid 4 to mira0647520-26 2019-10-01T02:25:01.083 INFO:tasks.rados.rados.0.mira064.stdout:26: finishing write tid 5 to mira0647520-26 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:30: finishing write tid 1 to mira0647520-30 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:27: finishing write tid 2 to mira0647520-27 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:27: finishing write tid 4 to mira0647520-27 2019-10-01T02:25:01.084 INFO:tasks.rados.rados.0.mira064.stdout:27: finishing write tid 5 to mira0647520-27 2019-10-01T02:25:01.085 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2019-10-01T02:25:01.085 INFO:tasks.rados.rados.0.mira064.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2019-10-01T02:25:01.085 INFO:tasks.rados.rados.0.mira064.stdout:28: finishing write tid 1 to mira0647520-28 2019-10-01T02:25:01.085 INFO:tasks.rados.rados.0.mira064.stdout:26: done (15 left) 2019-10-01T02:25:01.085 INFO:tasks.rados.rados.0.mira064.stdout:27: done (14 left) 2019-10-01T02:25:01.086 INFO:tasks.rados.rados.0.mira064.stdout:42: write initial oid 42 2019-10-01T02:25:01.086 INFO:tasks.rados.rados.0.mira064.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2019-10-01T02:25:01.095 INFO:tasks.rados.rados.0.mira064.stdout:42: writing mira0647520-42 from 689087 to 1156061 tid 1 2019-10-01T02:25:01.108 INFO:tasks.rados.rados.0.mira064.stdout:42: writing mira0647520-42 from 1699644 to 2483976 tid 2 2019-10-01T02:25:01.108 INFO:tasks.rados.rados.0.mira064.stdout:43: write initial oid 43 2019-10-01T02:25:01.114 INFO:tasks.rados.rados.0.mira064.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2019-10-01T02:25:01.119 INFO:tasks.rados.rados.0.mira064.stdout:43: writing mira0647520-43 from 705894 to 1248117 tid 1 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout:43: writing mira0647520-43 from 1758126 to 2402469 tid 2 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout:29: finishing write tid 1 to mira0647520-29 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout:31: finishing write tid 1 to mira0647520-31 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout:31: finishing write tid 2 to mira0647520-31 2019-10-01T02:25:01.131 INFO:tasks.rados.rados.0.mira064.stdout:31: finishing write tid 4 to mira0647520-31 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:31: finishing write tid 5 to mira0647520-31 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:31: done (15 left) 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:44: write initial oid 44 2019-10-01T02:25:01.132 INFO:tasks.rados.rados.0.mira064.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2019-10-01T02:25:01.143 INFO:tasks.rados.rados.0.mira064.stdout:44: writing mira0647520-44 from 722701 to 1340173 tid 1 2019-10-01T02:25:01.155 INFO:tasks.rados.rados.0.mira064.stdout:44: writing mira0647520-44 from 1816608 to 2604609 tid 2 2019-10-01T02:25:01.156 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:32: finishing write tid 1 to mira0647520-32 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:30: finishing write tid 2 to mira0647520-30 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:30: finishing write tid 4 to mira0647520-30 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:30: finishing write tid 5 to mira0647520-30 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 30 v 24 (ObjNum 29 snap 0 seq_num 29) dirty exists 2019-10-01T02:25:01.157 INFO:tasks.rados.rados.0.mira064.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:28: finishing write tid 2 to mira0647520-28 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:28: finishing write tid 4 to mira0647520-28 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:28: finishing write tid 5 to mira0647520-28 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:29: finishing write tid 2 to mira0647520-29 2019-10-01T02:25:01.158 INFO:tasks.rados.rados.0.mira064.stdout:28: done (15 left) 2019-10-01T02:25:01.159 INFO:tasks.rados.rados.0.mira064.stdout:30: done (14 left) 2019-10-01T02:25:01.159 INFO:tasks.rados.rados.0.mira064.stdout:45: write initial oid 45 2019-10-01T02:25:01.159 INFO:tasks.rados.rados.0.mira064.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2019-10-01T02:25:01.173 INFO:tasks.rados.rados.0.mira064.stdout:45: writing mira0647520-45 from 739508 to 1432229 tid 1 2019-10-01T02:25:01.180 INFO:tasks.rados.rados.0.mira064.stdout:45: writing mira0647520-45 from 1875090 to 2523102 tid 2 2019-10-01T02:25:01.180 INFO:tasks.rados.rados.0.mira064.stdout:46: write initial oid 46 2019-10-01T02:25:01.184 INFO:tasks.rados.rados.0.mira064.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2019-10-01T02:25:01.195 INFO:tasks.rados.rados.0.mira064.stdout:46: writing mira0647520-46 from 756315 to 1524285 tid 1 2019-10-01T02:25:01.206 INFO:tasks.rados.rados.0.mira064.stdout:46: writing mira0647520-46 from 1933572 to 2725242 tid 2 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout:29: finishing write tid 4 to mira0647520-29 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout:29: finishing write tid 5 to mira0647520-29 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2019-10-01T02:25:01.207 INFO:tasks.rados.rados.0.mira064.stdout:32: finishing write tid 2 to mira0647520-32 2019-10-01T02:25:01.208 INFO:tasks.rados.rados.0.mira064.stdout:32: finishing write tid 4 to mira0647520-32 2019-10-01T02:25:01.208 INFO:tasks.rados.rados.0.mira064.stdout:32: finishing write tid 5 to mira0647520-32 2019-10-01T02:25:01.208 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:34: finishing write tid 1 to mira0647520-34 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:34: finishing write tid 2 to mira0647520-34 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:29: done (15 left) 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:32: done (14 left) 2019-10-01T02:25:01.209 INFO:tasks.rados.rados.0.mira064.stdout:47: write initial oid 47 2019-10-01T02:25:01.213 INFO:tasks.rados.rados.0.mira064.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2019-10-01T02:25:01.219 INFO:tasks.rados.rados.0.mira064.stdout:47: writing mira0647520-47 from 773122 to 1332694 tid 1 2019-10-01T02:25:01.232 INFO:tasks.rados.rados.0.mira064.stdout:47: writing mira0647520-47 from 1992054 to 2643735 tid 2 2019-10-01T02:25:01.232 INFO:tasks.rados.rados.0.mira064.stdout:48: write initial oid 48 2019-10-01T02:25:01.232 INFO:tasks.rados.rados.0.mira064.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2019-10-01T02:25:01.245 INFO:tasks.rados.rados.0.mira064.stdout:48: writing mira0647520-48 from 789929 to 1424750 tid 1 2019-10-01T02:25:01.259 INFO:tasks.rados.rados.0.mira064.stdout:48: writing mira0647520-48 from 2050536 to 2789929 tid 2 2019-10-01T02:25:01.260 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.260 INFO:tasks.rados.rados.0.mira064.stdout:34: finishing write tid 4 to mira0647520-34 2019-10-01T02:25:01.260 INFO:tasks.rados.rados.0.mira064.stdout:34: finishing write tid 5 to mira0647520-34 2019-10-01T02:25:01.260 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2019-10-01T02:25:01.261 INFO:tasks.rados.rados.0.mira064.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2019-10-01T02:25:01.261 INFO:tasks.rados.rados.0.mira064.stdout:35: finishing write tid 1 to mira0647520-35 2019-10-01T02:25:01.261 INFO:tasks.rados.rados.0.mira064.stdout:34: done (15 left) 2019-10-01T02:25:01.261 INFO:tasks.rados.rados.0.mira064.stdout:49: write initial oid 49 2019-10-01T02:25:01.262 INFO:tasks.rados.rados.0.mira064.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2019-10-01T02:25:01.269 INFO:tasks.rados.rados.0.mira064.stdout:49: writing mira0647520-49 from 406736 to 1116806 tid 1 2019-10-01T02:25:01.281 INFO:tasks.rados.rados.0.mira064.stdout:49: writing mira0647520-49 from 1709018 to 2364368 tid 2 2019-10-01T02:25:01.282 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.282 INFO:tasks.rados.rados.0.mira064.stdout:35: finishing write tid 2 to mira0647520-35 2019-10-01T02:25:01.282 INFO:tasks.rados.rados.0.mira064.stdout:35: finishing write tid 4 to mira0647520-35 2019-10-01T02:25:01.282 INFO:tasks.rados.rados.0.mira064.stdout:35: finishing write tid 5 to mira0647520-35 2019-10-01T02:25:01.282 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2019-10-01T02:25:01.283 INFO:tasks.rados.rados.0.mira064.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2019-10-01T02:25:01.283 INFO:tasks.rados.rados.0.mira064.stdout:40: finishing write tid 1 to mira0647520-40 2019-10-01T02:25:01.283 INFO:tasks.rados.rados.0.mira064.stdout:35: done (15 left) 2019-10-01T02:25:01.283 INFO:tasks.rados.rados.0.mira064.stdout:50: write initial oid 50 2019-10-01T02:25:01.283 INFO:tasks.rados.rados.0.mira064.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2019-10-01T02:25:01.292 INFO:tasks.rados.rados.0.mira064.stdout:50: writing mira0647520-50 from 423543 to 1208862 tid 1 2019-10-01T02:25:01.307 INFO:tasks.rados.rados.0.mira064.stdout:50: writing mira0647520-50 from 1767500 to 2566508 tid 2 2019-10-01T02:25:01.307 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.307 INFO:tasks.rados.rados.0.mira064.stdout:40: finishing write tid 2 to mira0647520-40 2019-10-01T02:25:01.307 INFO:tasks.rados.rados.0.mira064.stdout:40: finishing write tid 4 to mira0647520-40 2019-10-01T02:25:01.307 INFO:tasks.rados.rados.0.mira064.stdout:40: finishing write tid 5 to mira0647520-40 2019-10-01T02:25:01.308 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:40: done (15 left) 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:51: delete oid 17 current snap is 0 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:43: finishing write tid 1 to mira0647520-43 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:43: finishing write tid 2 to mira0647520-43 2019-10-01T02:25:01.309 INFO:tasks.rados.rados.0.mira064.stdout:43: finishing write tid 4 to mira0647520-43 2019-10-01T02:25:01.310 INFO:tasks.rados.rados.0.mira064.stdout:43: finishing write tid 5 to mira0647520-43 2019-10-01T02:25:01.310 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2019-10-01T02:25:01.310 INFO:tasks.rados.rados.0.mira064.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2019-10-01T02:25:01.310 INFO:tasks.rados.rados.0.mira064.stdout:48: finishing write tid 1 to mira0647520-48 2019-10-01T02:25:01.356 INFO:tasks.rados.rados.0.mira064.stdout:48: finishing write tid 2 to mira0647520-48 2019-10-01T02:25:01.356 INFO:tasks.rados.rados.0.mira064.stdout:48: finishing write tid 4 to mira0647520-48 2019-10-01T02:25:01.357 INFO:tasks.rados.rados.0.mira064.stdout:48: finishing write tid 5 to mira0647520-48 2019-10-01T02:25:01.357 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 48 v 3 (ObjNum 47 snap 0 seq_num 47) dirty exists 2019-10-01T02:25:01.357 INFO:tasks.rados.rados.0.mira064.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2019-10-01T02:25:01.357 INFO:tasks.rados.rados.0.mira064.stdout:49: finishing write tid 1 to mira0647520-49 2019-10-01T02:25:01.362 INFO:tasks.rados.rados.0.mira064.stdout:49: finishing write tid 2 to mira0647520-49 2019-10-01T02:25:01.362 INFO:tasks.rados.rados.0.mira064.stdout:49: finishing write tid 4 to mira0647520-49 2019-10-01T02:25:01.363 INFO:tasks.rados.rados.0.mira064.stdout:49: finishing write tid 5 to mira0647520-49 2019-10-01T02:25:01.363 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2019-10-01T02:25:01.363 INFO:tasks.rados.rados.0.mira064.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2019-10-01T02:25:01.533 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":6,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:25:01.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:25:01.558 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:25:01.760 INFO:tasks.rados.rados.0.mira064.stdout:33: finishing write tid 1 to mira0647520-33 2019-10-01T02:25:01.760 INFO:tasks.rados.rados.0.mira064.stdout:37: finishing write tid 1 to mira0647520-37 2019-10-01T02:25:01.761 INFO:tasks.rados.rados.0.mira064.stdout:39: finishing write tid 1 to mira0647520-39 2019-10-01T02:25:01.762 INFO:tasks.rados.rados.0.mira064.stdout:43: done (15 left) 2019-10-01T02:25:01.762 INFO:tasks.rados.rados.0.mira064.stdout:48: done (14 left) 2019-10-01T02:25:01.762 INFO:tasks.rados.rados.0.mira064.stdout:49: done (13 left) 2019-10-01T02:25:01.763 INFO:tasks.rados.rados.0.mira064.stdout:51: done (12 left) 2019-10-01T02:25:01.763 INFO:tasks.rados.rados.0.mira064.stdout:52: write oid 12 current snap is 0 2019-10-01T02:25:01.763 INFO:tasks.rados.rados.0.mira064.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2019-10-01T02:25:01.780 INFO:tasks.rados.rados.0.mira064.stdout:52: writing mira0647520-12 from 457157 to 992974 tid 1 2019-10-01T02:25:01.796 INFO:tasks.rados.rados.0.mira064.stdout:52: writing mira0647520-12 from 1768111 to 2170788 tid 2 2019-10-01T02:25:01.796 INFO:tasks.rados.rados.0.mira064.stdout:38: finishing write tid 1 to mira0647520-38 2019-10-01T02:25:01.797 INFO:tasks.rados.rados.0.mira064.stdout:36: finishing write tid 1 to mira0647520-36 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:42: finishing write tid 1 to mira0647520-42 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:45: finishing write tid 1 to mira0647520-45 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:41: finishing write tid 1 to mira0647520-41 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:53: read oid 6 snap -1 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:53: expect (ObjNum 5 snap 0 seq_num 5) 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:54: write (excl) oid 34 current snap is 0 2019-10-01T02:25:01.801 INFO:tasks.rados.rados.0.mira064.stdout:54: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2019-10-01T02:25:01.811 INFO:tasks.rados.rados.0.mira064.stdout:54: writing mira0647520-34 from 473964 to 1085030 tid 1 2019-10-01T02:25:01.827 INFO:tasks.rados.rados.0.mira064.stdout:54: writing mira0647520-34 from 1826593 to 2372928 tid 2 2019-10-01T02:25:01.839 INFO:tasks.rados.rados.0.mira064.stdout:54: writing mira0647520-34 from 2778819 to 2873964 tid 3 2019-10-01T02:25:01.839 INFO:tasks.rados.rados.0.mira064.stdout:55: copy_from oid 20 from oid 2 current snap is 0 2019-10-01T02:25:01.839 INFO:tasks.rados.rados.0.mira064.stdout:50: finishing write tid 1 to mira0647520-50 2019-10-01T02:25:01.839 INFO:tasks.rados.rados.0.mira064.stdout:44: finishing write tid 1 to mira0647520-44 2019-10-01T02:25:01.840 INFO:tasks.rados.rados.0.mira064.stdout:46: finishing write tid 1 to mira0647520-46 2019-10-01T02:25:01.840 INFO:tasks.rados.rados.0.mira064.stdout:47: finishing write tid 1 to mira0647520-47 2019-10-01T02:25:01.840 INFO:tasks.rados.rados.0.mira064.stdout:33: finishing write tid 2 to mira0647520-33 2019-10-01T02:25:01.840 INFO:tasks.rados.rados.0.mira064.stdout:36: finishing write tid 2 to mira0647520-36 2019-10-01T02:25:01.841 INFO:tasks.rados.rados.0.mira064.stdout: waiting on 16 2019-10-01T02:25:01.841 INFO:tasks.rados.rados.0.mira064.stdout:36: finishing write tid 4 to mira0647520-36 2019-10-01T02:25:01.841 INFO:tasks.rados.rados.0.mira064.stdout:36: finishing write tid 5 to mira0647520-36 2019-10-01T02:25:01.841 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2019-10-01T02:25:01.841 INFO:tasks.rados.rados.0.mira064.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2019-10-01T02:25:01.842 INFO:tasks.rados.rados.0.mira064.stdout:38: finishing write tid 2 to mira0647520-38 2019-10-01T02:25:01.842 INFO:tasks.rados.rados.0.mira064.stdout:38: finishing write tid 4 to mira0647520-38 2019-10-01T02:25:01.842 INFO:tasks.rados.rados.0.mira064.stdout:33: finishing write tid 4 to mira0647520-33 2019-10-01T02:25:01.842 INFO:tasks.rados.rados.0.mira064.stdout:33: finishing write tid 5 to mira0647520-33 2019-10-01T02:25:01.843 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2019-10-01T02:25:01.843 INFO:tasks.rados.rados.0.mira064.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2019-10-01T02:25:01.843 INFO:tasks.rados.rados.0.mira064.stdout:38: finishing write tid 5 to mira0647520-38 2019-10-01T02:25:01.843 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 38 v 5 (ObjNum 37 snap 0 seq_num 37) dirty exists 2019-10-01T02:25:01.843 INFO:tasks.rados.rados.0.mira064.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2019-10-01T02:25:01.844 INFO:tasks.rados.rados.0.mira064.stdout:37: finishing write tid 2 to mira0647520-37 2019-10-01T02:25:01.844 INFO:tasks.rados.rados.0.mira064.stdout:37: finishing write tid 4 to mira0647520-37 2019-10-01T02:25:01.845 INFO:tasks.rados.rados.0.mira064.stdout:44: finishing write tid 2 to mira0647520-44 2019-10-01T02:25:01.845 INFO:tasks.rados.rados.0.mira064.stdout:44: finishing write tid 4 to mira0647520-44 2019-10-01T02:25:01.845 INFO:tasks.rados.rados.0.mira064.stdout:44: finishing write tid 5 to mira0647520-44 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:37: finishing write tid 5 to mira0647520-37 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 37 v 29 (ObjNum 36 snap 0 seq_num 36) dirty exists 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:39: finishing write tid 2 to mira0647520-39 2019-10-01T02:25:01.846 INFO:tasks.rados.rados.0.mira064.stdout:39: finishing write tid 4 to mira0647520-39 2019-10-01T02:25:01.847 INFO:tasks.rados.rados.0.mira064.stdout:39: finishing write tid 5 to mira0647520-39 2019-10-01T02:25:01.847 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2019-10-01T02:25:01.847 INFO:tasks.rados.rados.0.mira064.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2019-10-01T02:25:01.847 INFO:tasks.rados.rados.0.mira064.stdout:46: finishing write tid 2 to mira0647520-46 2019-10-01T02:25:01.848 INFO:tasks.rados.rados.0.mira064.stdout:45: finishing write tid 2 to mira0647520-45 2019-10-01T02:25:01.848 INFO:tasks.rados.rados.0.mira064.stdout:46: finishing write tid 4 to mira0647520-46 2019-10-01T02:25:01.848 INFO:tasks.rados.rados.0.mira064.stdout:45: finishing write tid 4 to mira0647520-45 2019-10-01T02:25:01.848 INFO:tasks.rados.rados.0.mira064.stdout:46: finishing write tid 5 to mira0647520-46 2019-10-01T02:25:01.849 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 46 v 15 (ObjNum 45 snap 0 seq_num 45) dirty exists 2019-10-01T02:25:01.849 INFO:tasks.rados.rados.0.mira064.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2019-10-01T02:25:01.849 INFO:tasks.rados.rados.0.mira064.stdout:45: finishing write tid 5 to mira0647520-45 2019-10-01T02:25:01.849 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 45 v 31 (ObjNum 44 snap 0 seq_num 44) dirty exists 2019-10-01T02:25:01.849 INFO:tasks.rados.rados.0.mira064.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:33: done (15 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:36: done (14 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:37: done (13 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:38: done (12 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:39: done (11 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:44: done (10 left) 2019-10-01T02:25:01.850 INFO:tasks.rados.rados.0.mira064.stdout:45: done (9 left) 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:46: done (8 left) 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:56: copy_from oid 38 from oid 17 current snap is 0 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:41: finishing write tid 2 to mira0647520-41 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:42: finishing write tid 2 to mira0647520-42 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:41: finishing write tid 4 to mira0647520-41 2019-10-01T02:25:01.851 INFO:tasks.rados.rados.0.mira064.stdout:41: finishing write tid 5 to mira0647520-41 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:42: finishing write tid 4 to mira0647520-42 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:42: finishing write tid 5 to mira0647520-42 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 42 v 33 (ObjNum 41 snap 0 seq_num 41) dirty exists 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2019-10-01T02:25:01.852 INFO:tasks.rados.rados.0.mira064.stdout:41: done (8 left) 2019-10-01T02:25:01.853 INFO:tasks.rados.rados.0.mira064.stdout:42: done (7 left) 2019-10-01T02:25:01.853 INFO:tasks.rados.rados.0.mira064.stdout:57: read oid 11 snap -1 2019-10-01T02:25:01.853 INFO:tasks.rados.rados.0.mira064.stdout:57: expect (ObjNum 10 snap 0 seq_num 10) 2019-10-01T02:25:01.853 INFO:tasks.rados.rados.0.mira064.stdout:58: snap_create 2019-10-01T02:25:01.853 INFO:tasks.rados.rados.0.mira064.stdout:53: expect (ObjNum 5 snap 0 seq_num 5) 2019-10-01T02:25:02.018 INFO:tasks.rados.rados.0.mira064.stdout:47: finishing write tid 2 to mira0647520-47 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:50: finishing write tid 2 to mira0647520-50 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:50: finishing write tid 4 to mira0647520-50 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:47: finishing write tid 4 to mira0647520-47 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:47: finishing write tid 5 to mira0647520-47 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2019-10-01T02:25:02.019 INFO:tasks.rados.rados.0.mira064.stdout:50: finishing write tid 5 to mira0647520-50 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:52: finishing write tid 1 to mira0647520-12 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:52: finishing write tid 2 to mira0647520-12 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:52: finishing write tid 4 to mira0647520-12 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:52: finishing write tid 5 to mira0647520-12 2019-10-01T02:25:02.020 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 12 v 36 (ObjNum 51 snap 0 seq_num 51) dirty exists 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:52: left oid 12 (ObjNum 51 snap 0 seq_num 51) 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:54: finishing write tid 1 to mira0647520-34 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:54: finishing write tid 2 to mira0647520-34 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:54: finishing write tid 3 to mira0647520-34 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:54: finishing write tid 5 to mira0647520-34 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:54: finishing write tid 6 to mira0647520-34 2019-10-01T02:25:02.021 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 34 v 19 (ObjNum 52 snap 0 seq_num 52) dirty exists 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:54: left oid 34 (ObjNum 52 snap 0 seq_num 52) 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:56: finishing copy_from to mira0647520-38 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:56: got expected ENOENT (src dne) 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:56: finishing copy_from racing read to mira0647520-38 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:55: finishing copy_from to mira0647520-20 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 20 v 6 (ObjNum 1 snap 0 seq_num 1) dirty exists 2019-10-01T02:25:02.022 INFO:tasks.rados.rados.0.mira064.stdout:55: finishing copy_from racing read to mira0647520-20 2019-10-01T02:25:02.023 INFO:tasks.rados.rados.0.mira064.stdout:57: expect (ObjNum 10 snap 0 seq_num 10) 2019-10-01T02:25:02.565 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":6,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:25:02.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:25:02.796 INFO:tasks.rados.rados.0.mira064.stdout:47: done (8 left) 2019-10-01T02:25:02.796 INFO:tasks.rados.rados.0.mira064.stdout:50: done (7 left) 2019-10-01T02:25:02.797 INFO:tasks.rados.rados.0.mira064.stdout:52: done (6 left) 2019-10-01T02:25:02.797 INFO:tasks.rados.rados.0.mira064.stdout:53: done (5 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:54: done (4 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:55: done (3 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:56: done (2 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:57: done (1 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:58: done (0 left) 2019-10-01T02:25:02.798 INFO:tasks.rados.rados.0.mira064.stdout:59: read oid 11 snap 0 2019-10-01T02:25:02.799 INFO:tasks.rados.rados.0.mira064.stdout:59: expect (ObjNum 10 snap 0 seq_num 10) 2019-10-01T02:25:02.799 INFO:tasks.rados.rados.0.mira064.stdout:60: delete oid 25 current snap is 1 2019-10-01T02:25:02.802 INFO:tasks.rados.rados.0.mira064.stdout:60: done (1 left) 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:61: read oid 45 snap -1 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:61: expect (ObjNum 44 snap 0 seq_num 44) 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:62: read oid 44 snap -1 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:62: expect (ObjNum 43 snap 0 seq_num 43) 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:63: rollback oid 17 current snap is 1 2019-10-01T02:25:02.803 INFO:tasks.rados.rados.0.mira064.stdout:rollback oid 17 to 0 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:64: read oid 34 snap -1 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:64: expect (ObjNum 52 snap 0 seq_num 52) 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:65: rollback oid 39 current snap is 1 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:rollback oid 39 to 0 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:66: read oid 24 snap -1 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:66: expect (ObjNum 23 snap 0 seq_num 23) 2019-10-01T02:25:02.804 INFO:tasks.rados.rados.0.mira064.stdout:67: snap_create 2019-10-01T02:25:02.822 INFO:tasks.rados.rados.0.mira064.stdout:64: expect (ObjNum 52 snap 0 seq_num 52) 2019-10-01T02:25:03.129 INFO:tasks.rados.rados.0.mira064.stdout:59: expect (ObjNum 10 snap 0 seq_num 10) 2019-10-01T02:25:03.242 INFO:tasks.rados.rados.0.mira064.stdout:62: expect (ObjNum 43 snap 0 seq_num 43) 2019-10-01T02:25:04.944 INFO:tasks.rados.rados.0.mira064.stdout:66: expect (ObjNum 23 snap 0 seq_num 23) 2019-10-01T02:25:05.159 INFO:tasks.rados.rados.0.mira064.stdout:61: expect (ObjNum 44 snap 0 seq_num 44) 2019-10-01T02:25:09.721 INFO:tasks.rados.rados.0.mira064.stdout:63: finishing rollback tid 1 to mira0647520-17 2019-10-01T02:25:09.722 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 17 v 0 (ObjNum -52135992 snap 32671 seq_num 32) dirty dne 2019-10-01T02:25:09.722 INFO:tasks.rados.rados.0.mira064.stdout:65: finishing rollback tid 0 to mira0647520-39 2019-10-01T02:25:09.722 INFO:tasks.rados.rados.0.mira064.stdout:65: finishing rollback tid 1 to mira0647520-39 2019-10-01T02:25:09.722 INFO:tasks.rados.rados.0.mira064.stdout:65: finishing rollback tid 2 to mira0647520-39 2019-10-01T02:25:09.722 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 39 v 13 (ObjNum 38 snap 0 seq_num 38) dirty exists 2019-10-01T02:25:09.735 INFO:tasks.rados.rados.0.mira064.stdout:59: done (7 left) 2019-10-01T02:25:09.736 INFO:tasks.rados.rados.0.mira064.stdout:61: done (6 left) 2019-10-01T02:25:09.736 INFO:tasks.rados.rados.0.mira064.stdout:62: done (5 left) 2019-10-01T02:25:09.737 INFO:tasks.rados.rados.0.mira064.stdout:63: done (4 left) 2019-10-01T02:25:09.737 INFO:tasks.rados.rados.0.mira064.stdout:64: done (3 left) 2019-10-01T02:25:09.738 INFO:tasks.rados.rados.0.mira064.stdout:65: done (2 left) 2019-10-01T02:25:09.738 INFO:tasks.rados.rados.0.mira064.stdout:66: done (1 left) 2019-10-01T02:25:09.740 INFO:tasks.rados.rados.0.mira064.stdout:67: done (0 left) 2019-10-01T02:25:09.740 INFO:tasks.rados.rados.0.mira064.stdout:68: read oid 20 snap 0 2019-10-01T02:25:09.740 INFO:tasks.rados.rados.0.mira064.stdout:68: expect (ObjNum 1 snap 0 seq_num 1) 2019-10-01T02:25:09.740 INFO:tasks.rados.rados.0.mira064.stdout:69: snap_remove snap 1 2019-10-01T02:25:10.740 INFO:tasks.rados.rados.0.mira064.stdout:69: done (1 left) 2019-10-01T02:25:10.741 INFO:tasks.rados.rados.0.mira064.stdout:70: rollback oid 13 current snap is 2 2019-10-01T02:25:10.741 INFO:tasks.rados.rados.0.mira064.stdout:rollback oid 13 to 0 2019-10-01T02:25:10.741 INFO:tasks.rados.rados.0.mira064.stdout:68: expect (ObjNum 1 snap 0 seq_num 1) 2019-10-01T02:25:10.921 INFO:tasks.rados.rados.0.mira064.stdout:70: finishing rollback tid 0 to mira0647520-13 2019-10-01T02:25:10.921 INFO:tasks.rados.rados.0.mira064.stdout:70: finishing rollback tid 1 to mira0647520-13 2019-10-01T02:25:10.921 INFO:tasks.rados.rados.0.mira064.stdout:70: finishing rollback tid 2 to mira0647520-13 2019-10-01T02:25:10.921 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 13 v 16 (ObjNum 12 snap 0 seq_num 12) dirty exists 2019-10-01T02:25:10.922 INFO:tasks.rados.rados.0.mira064.stdout:68: done (1 left) 2019-10-01T02:25:10.922 INFO:tasks.rados.rados.0.mira064.stdout:70: done (0 left) 2019-10-01T02:25:10.922 INFO:tasks.rados.rados.0.mira064.stdout:71: snap_remove snap 0 2019-10-01T02:25:11.760 INFO:tasks.rados.rados.0.mira064.stdout:71: done (0 left) 2019-10-01T02:25:11.760 INFO:tasks.rados.rados.0.mira064.stdout:72: rollback oid 49 current snap is 2 2019-10-01T02:25:11.760 INFO:tasks.rados.rados.0.mira064.stdout:72: done (0 left) 2019-10-01T02:25:11.760 INFO:tasks.rados.rados.0.mira064.stdout:73: snap_create 2019-10-01T02:25:12.018 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:25:12.009+0000 7fed41a93700 -1 osd.1 pg_epoch: 23 pg[2.7( v 20'17 (0'0,20'17] local-lis/les=16/17 n=6 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [1,0] r=0 lpr=16 crt=20'17 lcod 19'15 mlcod 19'15 active+clean+snaptrim trimq=[2~1] ps=[3~1]] removing snap head 2019-10-01T02:25:12.764 INFO:tasks.rados.rados.0.mira064.stdout:73: done (0 left) 2019-10-01T02:25:12.764 INFO:tasks.rados.rados.0.mira064.stdout:74: snap_create 2019-10-01T02:25:13.784 INFO:tasks.rados.rados.0.mira064.stdout:74: done (0 left) 2019-10-01T02:25:13.784 INFO:tasks.rados.rados.0.mira064.stdout:75: delete oid 20 current snap is 4 2019-10-01T02:25:13.790 INFO:tasks.rados.rados.0.mira064.stdout:75: done (0 left) 2019-10-01T02:25:13.790 INFO:tasks.rados.rados.0.mira064.stdout:76: write (excl) oid 45 current snap is 4 2019-10-01T02:25:13.791 INFO:tasks.rados.rados.0.mira064.stdout:76: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2019-10-01T02:25:13.814 INFO:tasks.rados.rados.0.mira064.stdout:76: writing mira0647520-45 from 524385 to 1077551 tid 1 2019-10-01T02:25:13.841 INFO:tasks.rados.rados.0.mira064.stdout:76: writing mira0647520-45 from 1602039 to 2012054 tid 2 2019-10-01T02:25:13.851 INFO:tasks.rados.rados.0.mira064.stdout:76: writing mira0647520-45 from 2577441 to 2924385 tid 3 2019-10-01T02:25:13.851 INFO:tasks.rados.rados.0.mira064.stdout:77: copy_from oid 16 from oid 27 current snap is 4 2019-10-01T02:25:13.854 INFO:tasks.rados.rados.0.mira064.stdout:76: finishing write tid 1 to mira0647520-45 2019-10-01T02:25:13.854 INFO:tasks.rados.rados.0.mira064.stdout:76: finishing write tid 2 to mira0647520-45 2019-10-01T02:25:13.854 INFO:tasks.rados.rados.0.mira064.stdout:78: write (excl) oid 30 current snap is 4 2019-10-01T02:25:13.854 INFO:tasks.rados.rados.0.mira064.stdout:78: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2019-10-01T02:25:13.871 INFO:tasks.rados.rados.0.mira064.stdout:78: writing mira0647520-30 from 541192 to 1169607 tid 1 2019-10-01T02:25:13.888 INFO:tasks.rados.rados.0.mira064.stdout:78: writing mira0647520-30 from 1660521 to 2214194 tid 2 2019-10-01T02:25:13.899 INFO:tasks.rados.rados.0.mira064.stdout:78: writing mira0647520-30 from 2888511 to 2941192 tid 3 2019-10-01T02:25:13.900 INFO:tasks.rados.rados.0.mira064.stdout:79: read oid 3 snap -1 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:79: expect (ObjNum 2 snap 0 seq_num 2) 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:80: read oid 25 snap 2 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:80: expect deleted 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:81: read oid 12 snap 3 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:81: expect (ObjNum 51 snap 0 seq_num 51) 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:82: snap_create 2019-10-01T02:25:13.901 INFO:tasks.rados.rados.0.mira064.stdout:76: finishing write tid 3 to mira0647520-45 2019-10-01T02:25:13.902 INFO:tasks.rados.rados.0.mira064.stdout:76: finishing write tid 5 to mira0647520-45 2019-10-01T02:25:13.902 INFO:tasks.rados.rados.0.mira064.stdout:76: finishing write tid 6 to mira0647520-45 2019-10-01T02:25:13.902 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 45 v 41 (ObjNum 55 snap 4 seq_num 55) dirty exists 2019-10-01T02:25:13.902 INFO:tasks.rados.rados.0.mira064.stdout:76: left oid 45 (ObjNum 55 snap 4 seq_num 55) 2019-10-01T02:25:13.902 INFO:tasks.rados.rados.0.mira064.stdout:78: finishing write tid 1 to mira0647520-30 2019-10-01T02:25:13.929 INFO:tasks.rados.rados.0.mira064.stdout:77: finishing copy_from to mira0647520-16 2019-10-01T02:25:13.929 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 16 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2019-10-01T02:25:13.929 INFO:tasks.rados.rados.0.mira064.stdout:77: finishing copy_from racing read to mira0647520-16 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:78: finishing write tid 2 to mira0647520-30 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:78: finishing write tid 3 to mira0647520-30 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:78: finishing write tid 5 to mira0647520-30 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:78: finishing write tid 6 to mira0647520-30 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 30 v 46 (ObjNum 56 snap 4 seq_num 56) dirty exists 2019-10-01T02:25:13.930 INFO:tasks.rados.rados.0.mira064.stdout:78: left oid 30 (ObjNum 56 snap 4 seq_num 56) 2019-10-01T02:25:13.944 INFO:tasks.rados.rados.0.mira064.stdout:79: expect (ObjNum 2 snap 0 seq_num 2) 2019-10-01T02:25:14.131 INFO:tasks.rados.rados.0.mira064.stdout:81: expect (ObjNum 51 snap 0 seq_num 51) 2019-10-01T02:25:14.794 INFO:tasks.rados.rados.0.mira064.stdout:76: done (6 left) 2019-10-01T02:25:14.794 INFO:tasks.rados.rados.0.mira064.stdout:77: done (5 left) 2019-10-01T02:25:14.794 INFO:tasks.rados.rados.0.mira064.stdout:78: done (4 left) 2019-10-01T02:25:14.794 INFO:tasks.rados.rados.0.mira064.stdout:79: done (3 left) 2019-10-01T02:25:14.794 INFO:tasks.rados.rados.0.mira064.stdout:80: done (2 left) 2019-10-01T02:25:14.795 INFO:tasks.rados.rados.0.mira064.stdout:81: done (1 left) 2019-10-01T02:25:14.795 INFO:tasks.rados.rados.0.mira064.stdout:82: done (0 left) 2019-10-01T02:25:14.795 INFO:tasks.rados.rados.0.mira064.stdout:83: read oid 40 snap 4 2019-10-01T02:25:14.795 INFO:tasks.rados.rados.0.mira064.stdout:83: expect (ObjNum 39 snap 0 seq_num 39) 2019-10-01T02:25:14.796 INFO:tasks.rados.rados.0.mira064.stdout:84: delete oid 48 current snap is 5 2019-10-01T02:25:14.807 INFO:tasks.rados.rados.0.mira064.stdout:84: done (1 left) 2019-10-01T02:25:14.807 INFO:tasks.rados.rados.0.mira064.stdout:85: write (excl) oid 24 current snap is 5 2019-10-01T02:25:14.807 INFO:tasks.rados.rados.0.mira064.stdout:85: seq_num 58 ranges {574806=778913,2061132=557342} 2019-10-01T02:25:14.830 INFO:tasks.rados.rados.0.mira064.stdout:85: writing mira0647520-24 from 574806 to 1353719 tid 1 2019-10-01T02:25:14.849 INFO:tasks.rados.rados.0.mira064.stdout:85: writing mira0647520-24 from 2061132 to 2618474 tid 2 2019-10-01T02:25:14.850 INFO:tasks.rados.rados.0.mira064.stdout:86: read oid 1 snap 3 2019-10-01T02:25:14.850 INFO:tasks.rados.rados.0.mira064.stdout:86: expect (ObjNum 0 snap 0 seq_num 0) 2019-10-01T02:25:14.851 INFO:tasks.rados.rados.0.mira064.stdout:87: read oid 50 snap -1 2019-10-01T02:25:14.851 INFO:tasks.rados.rados.0.mira064.stdout:87: expect (ObjNum 49 snap 0 seq_num 49) 2019-10-01T02:25:14.851 INFO:tasks.rados.rados.0.mira064.stdout:88: copy_from oid 5 from oid 32 current snap is 5 2019-10-01T02:25:14.851 INFO:tasks.rados.rados.0.mira064.stdout:83: expect (ObjNum 39 snap 0 seq_num 39) 2019-10-01T02:25:15.011 INFO:tasks.rados.rados.0.mira064.stdout:85: finishing write tid 1 to mira0647520-24 2019-10-01T02:25:15.011 INFO:tasks.rados.rados.0.mira064.stdout:85: finishing write tid 2 to mira0647520-24 2019-10-01T02:25:15.011 INFO:tasks.rados.rados.0.mira064.stdout:85: finishing write tid 4 to mira0647520-24 2019-10-01T02:25:15.011 INFO:tasks.rados.rados.0.mira064.stdout:85: finishing write tid 5 to mira0647520-24 2019-10-01T02:25:15.012 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 24 v 10 (ObjNum 58 snap 5 seq_num 58) dirty exists 2019-10-01T02:25:15.012 INFO:tasks.rados.rados.0.mira064.stdout:85: left oid 24 (ObjNum 58 snap 5 seq_num 58) 2019-10-01T02:25:15.012 INFO:tasks.rados.rados.0.mira064.stdout:83: done (4 left) 2019-10-01T02:25:15.012 INFO:tasks.rados.rados.0.mira064.stdout:85: done (3 left) 2019-10-01T02:25:15.013 INFO:tasks.rados.rados.0.mira064.stdout:89: write oid 41 current snap is 5 2019-10-01T02:25:15.013 INFO:tasks.rados.rados.0.mira064.stdout:89: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2019-10-01T02:25:15.025 INFO:tasks.rados.rados.0.mira064.stdout:89: writing mira0647520-41 from 591613 to 1045775 tid 1 2019-10-01T02:25:15.041 INFO:tasks.rados.rados.0.mira064.stdout:89: writing mira0647520-41 from 1719614 to 2136967 tid 2 2019-10-01T02:25:15.053 INFO:tasks.rados.rados.0.mira064.stdout:89: writing mira0647520-41 from 2570780 to 2991613 tid 3 2019-10-01T02:25:15.053 INFO:tasks.rados.rados.0.mira064.stdout:90: snap_remove snap 4 2019-10-01T02:25:15.814 INFO:tasks.rados.rados.0.mira064.stdout:90: done (4 left) 2019-10-01T02:25:15.814 INFO:tasks.rados.rados.0.mira064.stdout:91: snap_remove snap 2 2019-10-01T02:25:16.824 INFO:tasks.rados.rados.0.mira064.stdout:91: done (4 left) 2019-10-01T02:25:16.824 INFO:tasks.rados.rados.0.mira064.stdout:92: snap_create 2019-10-01T02:25:16.824 INFO:tasks.rados.rados.0.mira064.stdout:86: expect (ObjNum 0 snap 0 seq_num 0) 2019-10-01T02:25:17.010 INFO:tasks.rados.rados.0.mira064.stdout:87: expect (ObjNum 49 snap 0 seq_num 49) 2019-10-01T02:25:17.160 INFO:tasks.rados.rados.0.mira064.stdout:88: finishing copy_from to mira0647520-5 2019-10-01T02:25:17.160 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 5 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2019-10-01T02:25:17.160 INFO:tasks.rados.rados.0.mira064.stdout:88: finishing copy_from racing read to mira0647520-5 2019-10-01T02:25:17.160 INFO:tasks.rados.rados.0.mira064.stdout:89: finishing write tid 1 to mira0647520-41 2019-10-01T02:25:17.160 INFO:tasks.rados.rados.0.mira064.stdout:89: finishing write tid 2 to mira0647520-41 2019-10-01T02:25:17.161 INFO:tasks.rados.rados.0.mira064.stdout:89: finishing write tid 3 to mira0647520-41 2019-10-01T02:25:17.161 INFO:tasks.rados.rados.0.mira064.stdout:89: finishing write tid 5 to mira0647520-41 2019-10-01T02:25:17.161 INFO:tasks.rados.rados.0.mira064.stdout:89: finishing write tid 6 to mira0647520-41 2019-10-01T02:25:17.161 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 41 v 14 (ObjNum 59 snap 5 seq_num 59) dirty exists 2019-10-01T02:25:17.161 INFO:tasks.rados.rados.0.mira064.stdout:89: left oid 41 (ObjNum 59 snap 5 seq_num 59) 2019-10-01T02:25:17.827 INFO:tasks.rados.rados.0.mira064.stdout:86: done (4 left) 2019-10-01T02:25:17.827 INFO:tasks.rados.rados.0.mira064.stdout:87: done (3 left) 2019-10-01T02:25:17.827 INFO:tasks.rados.rados.0.mira064.stdout:88: done (2 left) 2019-10-01T02:25:17.827 INFO:tasks.rados.rados.0.mira064.stdout:89: done (1 left) 2019-10-01T02:25:17.827 INFO:tasks.rados.rados.0.mira064.stdout:92: done (0 left) 2019-10-01T02:25:17.828 INFO:tasks.rados.rados.0.mira064.stdout:93: delete oid 11 current snap is 6 2019-10-01T02:25:17.834 INFO:tasks.rados.rados.0.mira064.stdout:93: done (0 left) 2019-10-01T02:25:17.834 INFO:tasks.rados.rados.0.mira064.stdout:94: rollback oid 5 current snap is 6 2019-10-01T02:25:17.834 INFO:tasks.rados.rados.0.mira064.stdout:rollback oid 5 to 3 2019-10-01T02:25:17.834 INFO:tasks.rados.rados.0.mira064.stdout:95: copy_from oid 14 from oid 3 current snap is 6 2019-10-01T02:25:17.835 INFO:tasks.rados.rados.0.mira064.stdout:96: rollback oid 8 current snap is 6 2019-10-01T02:25:17.835 INFO:tasks.rados.rados.0.mira064.stdout:rollback oid 8 to 3 2019-10-01T02:25:17.835 INFO:tasks.rados.rados.0.mira064.stdout:97: read oid 25 snap -1 2019-10-01T02:25:17.835 INFO:tasks.rados.rados.0.mira064.stdout:97: expect deleted 2019-10-01T02:25:17.835 INFO:tasks.rados.rados.0.mira064.stdout:98: snap_remove snap 5 2019-10-01T02:25:18.841 INFO:tasks.rados.rados.0.mira064.stdout:98: done (4 left) 2019-10-01T02:25:18.841 INFO:tasks.rados.rados.0.mira064.stdout:99: write (excl) oid 23 current snap is 6 2019-10-01T02:25:18.842 INFO:tasks.rados.rados.0.mira064.stdout:99: seq_num 61 ranges {625227=721013,1952931=704669} 2019-10-01T02:25:18.872 INFO:tasks.rados.rados.0.mira064.stdout:99: writing mira0647520-23 from 625227 to 1346240 tid 1 2019-10-01T02:25:18.899 INFO:tasks.rados.rados.0.mira064.stdout:99: writing mira0647520-23 from 1952931 to 2657600 tid 2 2019-10-01T02:25:18.899 INFO:tasks.rados.rados.0.mira064.stdout:100: snap_create 2019-10-01T02:25:18.899 INFO:tasks.rados.rados.0.mira064.stdout:94: finishing rollback tid 0 to mira0647520-5 2019-10-01T02:25:18.899 INFO:tasks.rados.rados.0.mira064.stdout:94: finishing rollback tid 1 to mira0647520-5 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:94: finishing rollback tid 2 to mira0647520-5 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 5 v 54 (ObjNum 4 snap 0 seq_num 4) dirty exists 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:96: finishing rollback tid 0 to mira0647520-8 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:96: finishing rollback tid 1 to mira0647520-8 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:96: finishing rollback tid 2 to mira0647520-8 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 8 v 58 (ObjNum 7 snap 0 seq_num 7) dirty exists 2019-10-01T02:25:18.900 INFO:tasks.rados.rados.0.mira064.stdout:95: finishing copy_from to mira0647520-14 2019-10-01T02:25:18.901 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 14 v 9 (ObjNum 2 snap 0 seq_num 2) dirty exists 2019-10-01T02:25:18.901 INFO:tasks.rados.rados.0.mira064.stdout:95: finishing copy_from racing read to mira0647520-14 2019-10-01T02:25:18.901 INFO:tasks.rados.rados.0.mira064.stdout:99: finishing write tid 1 to mira0647520-23 2019-10-01T02:25:18.906 INFO:tasks.rados.rados.0.mira064.stdout:99: finishing write tid 2 to mira0647520-23 2019-10-01T02:25:18.906 INFO:tasks.rados.rados.0.mira064.stdout:99: finishing write tid 4 to mira0647520-23 2019-10-01T02:25:18.907 INFO:tasks.rados.rados.0.mira064.stdout:99: finishing write tid 5 to mira0647520-23 2019-10-01T02:25:18.907 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 23 v 22 (ObjNum 61 snap 6 seq_num 61) dirty exists 2019-10-01T02:25:18.907 INFO:tasks.rados.rados.0.mira064.stdout:99: left oid 23 (ObjNum 61 snap 6 seq_num 61) 2019-10-01T02:25:19.740 INFO:tasks.rados.rados.0.mira064.stdout:94: done (5 left) 2019-10-01T02:25:19.740 INFO:tasks.rados.rados.0.mira064.stdout:95: done (4 left) 2019-10-01T02:25:19.740 INFO:tasks.rados.rados.0.mira064.stdout:96: done (3 left) 2019-10-01T02:25:19.741 INFO:tasks.rados.rados.0.mira064.stdout:97: done (2 left) 2019-10-01T02:25:19.741 INFO:tasks.rados.rados.0.mira064.stdout:99: done (1 left) 2019-10-01T02:25:19.741 INFO:tasks.rados.rados.0.mira064.stdout:100: done (0 left) 2019-10-01T02:25:19.741 INFO:tasks.rados.rados.0.mira064.stdout:101: delete oid 9 current snap is 7 2019-10-01T02:25:19.755 INFO:tasks.rados.rados.0.mira064.stdout:101: done (0 left) 2019-10-01T02:25:19.755 INFO:tasks.rados.rados.0.mira064.stdout:102: copy_from oid 2 from oid 14 current snap is 7 2019-10-01T02:25:19.755 INFO:tasks.rados.rados.0.mira064.stdout:103: write (excl) oid 26 current snap is 7 2019-10-01T02:25:19.755 INFO:tasks.rados.rados.0.mira064.stdout:103: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2019-10-01T02:25:19.781 INFO:tasks.rados.rados.0.mira064.stdout:103: writing mira0647520-26 from 658841 to 1130352 tid 1 2019-10-01T02:25:19.805 INFO:tasks.rados.rados.0.mira064.stdout:103: writing mira0647520-26 from 1553542 to 2261880 tid 2 2019-10-01T02:25:19.820 INFO:tasks.rados.rados.0.mira064.stdout:103: writing mira0647520-26 from 2964119 to 3058841 tid 3 2019-10-01T02:25:19.821 INFO:tasks.rados.rados.0.mira064.stdout:104: delete oid 1 current snap is 7 2019-10-01T02:25:19.821 INFO:tasks.rados.rados.0.mira064.stdout:103: finishing write tid 1 to mira0647520-26 2019-10-01T02:25:19.821 INFO:tasks.rados.rados.0.mira064.stdout:102: finishing copy_from to mira0647520-2 2019-10-01T02:25:19.821 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 2 v 9 (ObjNum 2 snap 0 seq_num 2) dirty exists 2019-10-01T02:25:19.822 INFO:tasks.rados.rados.0.mira064.stdout:102: finishing copy_from racing read to mira0647520-2 2019-10-01T02:25:19.828 INFO:tasks.rados.rados.0.mira064.stdout:102: done (2 left) 2019-10-01T02:25:19.828 INFO:tasks.rados.rados.0.mira064.stdout:104: done (1 left) 2019-10-01T02:25:19.828 INFO:tasks.rados.rados.0.mira064.stdout:105: snap_remove snap 6 2019-10-01T02:25:20.751 INFO:tasks.rados.rados.0.mira064.stdout:105: done (1 left) 2019-10-01T02:25:20.752 INFO:tasks.rados.rados.0.mira064.stdout:106: read oid 21 snap -1 2019-10-01T02:25:20.752 INFO:tasks.rados.rados.0.mira064.stdout:106: expect (ObjNum 20 snap 0 seq_num 20) 2019-10-01T02:25:20.752 INFO:tasks.rados.rados.0.mira064.stdout:107: read oid 10 snap -1 2019-10-01T02:25:20.752 INFO:tasks.rados.rados.0.mira064.stdout:107: expect (ObjNum 9 snap 0 seq_num 9) 2019-10-01T02:25:20.752 INFO:tasks.rados.rados.0.mira064.stdout:108: snap_remove snap 3 2019-10-01T02:25:21.756 INFO:tasks.rados.rados.0.mira064.stdout:108: done (3 left) 2019-10-01T02:25:21.756 INFO:tasks.rados.rados.0.mira064.stdout:109: write (excl) oid 23 current snap is 7 2019-10-01T02:25:21.756 INFO:tasks.rados.rados.0.mira064.stdout:109: seq_num 65 ranges {692455=622009,2070506=712007} 2019-10-01T02:25:21.785 INFO:tasks.rados.rados.0.mira064.stdout:109: writing mira0647520-23 from 692455 to 1314464 tid 1 2019-10-01T02:25:21.793 INFO:tasks.ceph.osd.3.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7fc9e528c700 time 2019-10-01T02:25:21.790740+0000 2019-10-01T02:25:21.793 INFO:tasks.ceph.osd.3.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.794 INFO:tasks.ceph.osd.5.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7f4c5f3f0700 time 2019-10-01T02:25:21.791839+0000 2019-10-01T02:25:21.794 INFO:tasks.ceph.osd.5.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.808 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.809 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5608f21ee323] 2019-10-01T02:25:21.809 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.809 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.809 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.810 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.810 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.810 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.810 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.811 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.811 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.811 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.811 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.812 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.812 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.812 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.814 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.815 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.815 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.816 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.816 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.816 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.816 INFO:tasks.ceph.osd.3.mira064.stderr:2019-10-01T02:25:21.804+0000 7fc9e528c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7fc9e528c700 time 2019-10-01T02:25:21.790740+0000 2019-10-01T02:25:21.817 INFO:tasks.ceph.osd.3.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.817 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.818 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.818 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5608f21ee323] 2019-10-01T02:25:21.818 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.818 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.818 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.819 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.819 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.819 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.819 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.820 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.820 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.820 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.820 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.821 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.821 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.821 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.821 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.822 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.822 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.822 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.822 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.823 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.823 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.823 INFO:tasks.ceph.osd.3.mira064.stderr:*** Caught signal (Aborted) ** 2019-10-01T02:25:21.824 INFO:tasks.ceph.osd.3.mira064.stderr: in thread 7fc9e528c700 thread_name:tp_osd_tp 2019-10-01T02:25:21.825 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.825 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56292c179323] 2019-10-01T02:25:21.825 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.825 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.826 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.826 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.826 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.826 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.826 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.827 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.827 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.827 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.827 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.828 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.828 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.828 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.828 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.829 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.829 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.829 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.829 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.830 INFO:tasks.ceph.osd.5.mira064.stderr:2019-10-01T02:25:21.804+0000 7f4c5f3f0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7f4c5f3f0700 time 2019-10-01T02:25:21.791839+0000 2019-10-01T02:25:21.830 INFO:tasks.ceph.osd.5.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.830 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.830 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.830 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56292c179323] 2019-10-01T02:25:21.831 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.832 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.832 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.832 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.832 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.833 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.833 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.833 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.833 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.833 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.834 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.834 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.834 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.835 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.836 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.836 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.836 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.836 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.837 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.837 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.837 INFO:tasks.ceph.osd.5.mira064.stderr:*** Caught signal (Aborted) ** 2019-10-01T02:25:21.837 INFO:tasks.ceph.osd.5.mira064.stderr: in thread 7f4c5f3f0700 thread_name:tp_osd_tp 2019-10-01T02:25:21.840 INFO:tasks.ceph.osd.2.mira063.stderr:2019-10-01T02:25:21.821+0000 7f6400b27700 -1 osd.2 pg_epoch: 33 pg[2.0( v 31'19 (0'0,31'19] local-lis/les=16/17 n=5 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [2,5] r=0 lpr=16 crt=31'19 lcod 31'18 mlcod 31'18 active+clean+snaptrim trimq=[7~1,b~1,d~1] ps=[2~1,6~1,8~1]] removing snap head 2019-10-01T02:25:21.840 INFO:tasks.rados.rados.0.mira064.stdout:109: writing mira0647520-23 from 2070506 to 2782513 tid 2 2019-10-01T02:25:21.841 INFO:tasks.rados.rados.0.mira064.stdout:110: delete oid 13 current snap is 7 2019-10-01T02:25:21.841 INFO:tasks.rados.rados.0.mira064.stdout:103: finishing write tid 2 to mira0647520-26 2019-10-01T02:25:21.841 INFO:tasks.rados.rados.0.mira064.stdout:103: finishing write tid 3 to mira0647520-26 2019-10-01T02:25:21.841 INFO:tasks.rados.rados.0.mira064.stdout:103: finishing write tid 5 to mira0647520-26 2019-10-01T02:25:21.842 INFO:tasks.rados.rados.0.mira064.stdout:103: finishing write tid 6 to mira0647520-26 2019-10-01T02:25:21.842 INFO:tasks.rados.rados.0.mira064.stdout:update_object_version oid 26 v 19 (ObjNum 63 snap 7 seq_num 63) dirty exists 2019-10-01T02:25:21.842 INFO:tasks.rados.rados.0.mira064.stdout:103: left oid 26 (ObjNum 63 snap 7 seq_num 63) 2019-10-01T02:25:21.842 INFO:tasks.rados.rados.0.mira064.stdout:107: expect (ObjNum 9 snap 0 seq_num 9) 2019-10-01T02:25:21.843 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.843 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (()+0xf5d0) [0x7f4c816715d0] 2019-10-01T02:25:21.843 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (gsignal()+0x37) [0x7f4c804682c7] 2019-10-01T02:25:21.843 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (abort()+0x148) [0x7f4c804699b8] 2019-10-01T02:25:21.844 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56292c179372] 2019-10-01T02:25:21.844 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.844 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.845 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.845 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.846 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.846 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.846 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.847 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 22: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr: 23: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.848 INFO:tasks.ceph.osd.5.mira064.stderr:2019-10-01T02:25:21.820+0000 7f4c5f3f0700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: in thread 7f4c5f3f0700 thread_name:tp_osd_tp 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (()+0xf5d0) [0x7f4c816715d0] 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (gsignal()+0x37) [0x7f4c804682c7] 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (abort()+0x148) [0x7f4c804699b8] 2019-10-01T02:25:21.849 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56292c179372] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.850 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.851 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.852 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.852 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.852 INFO:tasks.ceph.osd.5.mira064.stderr: 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.852 INFO:tasks.ceph.osd.5.mira064.stderr: 22: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.852 INFO:tasks.ceph.osd.5.mira064.stderr: 23: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.853 INFO:tasks.ceph.osd.5.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.853 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.853 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.853 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (()+0xf5d0) [0x7fca0750d5d0] 2019-10-01T02:25:21.853 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (gsignal()+0x37) [0x7fca063042c7] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (abort()+0x148) [0x7fca063059b8] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x5608f21ee372] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.854 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.855 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.856 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.856 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.857 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: 23: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: 24: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr:2019-10-01T02:25:21.821+0000 7fc9e528c700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.859 INFO:tasks.ceph.osd.3.mira064.stderr: in thread 7fc9e528c700 thread_name:tp_osd_tp 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (()+0xf5d0) [0x7fca0750d5d0] 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (gsignal()+0x37) [0x7fca063042c7] 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (abort()+0x148) [0x7fca063059b8] 2019-10-01T02:25:21.860 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x5608f21ee372] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.861 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.862 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 23: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 24: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.863 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr: -13> 2019-10-01T02:25:21.804+0000 7f4c5f3f0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7f4c5f3f0700 time 2019-10-01T02:25:21.791839+0000 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56292c179323] 2019-10-01T02:25:21.868 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.869 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.869 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.869 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.869 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.869 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.870 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.871 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.872 INFO:tasks.ceph.osd.5.mira064.stderr: 0> 2019-10-01T02:25:21.820+0000 7f4c5f3f0700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.872 INFO:tasks.ceph.osd.5.mira064.stderr: in thread 7f4c5f3f0700 thread_name:tp_osd_tp 2019-10-01T02:25:21.872 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.872 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.873 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (()+0xf5d0) [0x7f4c816715d0] 2019-10-01T02:25:21.873 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (gsignal()+0x37) [0x7f4c804682c7] 2019-10-01T02:25:21.873 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (abort()+0x148) [0x7f4c804699b8] 2019-10-01T02:25:21.873 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56292c179372] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.874 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.875 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 22: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.876 INFO:tasks.ceph.osd.5.mira064.stderr: 23: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.877 INFO:tasks.ceph.osd.5.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.877 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.882 INFO:tasks.ceph.osd.3.mira064.stderr: -14> 2019-10-01T02:25:21.804+0000 7fc9e528c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7fc9e528c700 time 2019-10-01T02:25:21.790740+0000 2019-10-01T02:25:21.882 INFO:tasks.ceph.osd.3.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.882 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5608f21ee323] 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.883 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.884 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.885 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: 0> 2019-10-01T02:25:21.821+0000 7fc9e528c700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: in thread 7fc9e528c700 thread_name:tp_osd_tp 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.886 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (()+0xf5d0) [0x7fca0750d5d0] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (gsignal()+0x37) [0x7fca063042c7] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (abort()+0x148) [0x7fca063059b8] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x5608f21ee372] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.887 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.888 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.888 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.888 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.889 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 23: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: 24: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.890 INFO:tasks.ceph.osd.3.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.891 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.891 INFO:tasks.ceph.osd.5.mira064.stderr: -30> 2019-10-01T02:25:21.804+0000 7f4c5f3f0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7f4c5f3f0700 time 2019-10-01T02:25:21.791839+0000 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x56292c179323] 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.892 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.893 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.894 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: -17> 2019-10-01T02:25:21.820+0000 7f4c5f3f0700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: in thread 7f4c5f3f0700 thread_name:tp_osd_tp 2019-10-01T02:25:21.895 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 1: (()+0xf5d0) [0x7f4c816715d0] 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 2: (gsignal()+0x37) [0x7f4c804682c7] 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 3: (abort()+0x148) [0x7f4c804699b8] 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x56292c179372] 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 5: (()+0x4ea4eb) [0x56292c1794eb] 2019-10-01T02:25:21.896 INFO:tasks.ceph.osd.5.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x56292c724a15] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x56292c6de3d9] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x56292c6fafbb] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x56292c6fbbdb] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x56292c6ff08b] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x56292c71b12e] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x56292c3e3814] 2019-10-01T02:25:21.897 INFO:tasks.ceph.osd.5.mira064.stderr: 13: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc38) [0x56292c56ba58] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 14: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x257) [0x56292c57af87] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 15: (PGBackend::handle_message(boost::intrusive_ptr)+0x4a) [0x56292c41829a] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 16: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x5cb) [0x56292c3bd0cb] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 17: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x56292c25a229] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 18: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56292c4a0cb9] 2019-10-01T02:25:21.898 INFO:tasks.ceph.osd.5.mira064.stderr: 19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x56292c276702] 2019-10-01T02:25:21.899 INFO:tasks.ceph.osd.5.mira064.stderr: 20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x56292c84a616] 2019-10-01T02:25:21.899 INFO:tasks.ceph.osd.5.mira064.stderr: 21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56292c84d160] 2019-10-01T02:25:21.899 INFO:tasks.ceph.osd.5.mira064.stderr: 22: (()+0x7dd5) [0x7f4c81669dd5] 2019-10-01T02:25:21.900 INFO:tasks.ceph.osd.5.mira064.stderr: 23: (clone()+0x6d) [0x7f4c8053002d] 2019-10-01T02:25:21.900 INFO:tasks.ceph.osd.5.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.900 INFO:tasks.ceph.osd.5.mira064.stderr: 2019-10-01T02:25:21.927 INFO:tasks.ceph.osd.3.mira064.stderr: -15> 2019-10-01T02:25:21.804+0000 7fc9e528c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: In function 'auto bluestore_blob_t::get_unused_len() const' thread 7fc9e528c700 time 2019-10-01T02:25:21.790740+0000 2019-10-01T02:25:21.927 INFO:tasks.ceph.osd.3.mira064.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/15.0.0-5673-g2c0b216/rpm/el7/BUILD/ceph-15.0.0-5673-g2c0b216/src/os/bluestore/bluestore_types.h: 650: FAILED ceph_assert(extents.size() == 1 || extents.size() == 2) 2019-10-01T02:25:21.927 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.927 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5608f21ee323] 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.928 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.929 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.930 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.930 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.930 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.930 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: -1> 2019-10-01T02:25:21.821+0000 7fc9e528c700 -1 *** Caught signal (Aborted) ** 2019-10-01T02:25:21.931 INFO:tasks.ceph.osd.3.mira064.stderr: in thread 7fc9e528c700 thread_name:tp_osd_tp 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: ceph version 15.0.0-5673-g2c0b216 (2c0b2169f6303e11ab6bed736272e5e2d23ab027) octopus (dev) 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 1: (()+0xf5d0) [0x7fca0750d5d0] 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 2: (gsignal()+0x37) [0x7fca063042c7] 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 3: (abort()+0x148) [0x7fca063059b8] 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x5608f21ee372] 2019-10-01T02:25:21.932 INFO:tasks.ceph.osd.3.mira064.stderr: 5: (()+0x4ea4eb) [0x5608f21ee4eb] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 6: (bluestore_blob_t::mark_used(unsigned long, unsigned long)+0x1c5) [0x5608f2799a15] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 7: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr, BlueStore::WriteContext*)+0x1029) [0x5608f27533d9] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 8: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0x5bb) [0x5608f276ffbb] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 9: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v14_2_0::list&, unsigned int)+0xcb) [0x5608f2770bdb] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 10: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x166b) [0x5608f277408b] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 11: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x36e) [0x5608f279012e] 2019-10-01T02:25:21.933 INFO:tasks.ceph.osd.3.mira064.stderr: 12: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x54) [0x5608f2458814] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 13: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector > const&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xd18) [0x5608f25dd7a8] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 14: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xd10) [0x5608f23d5fa0] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfdd) [0x5608f2428fed] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x5608f242cc2f] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x5608f243278d] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x5608f22cf229] 2019-10-01T02:25:21.934 INFO:tasks.ceph.osd.3.mira064.stderr: 19: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5608f2515cb9] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1302) [0x5608f22eb702] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5608f28bf616] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5608f28c2160] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 23: (()+0x7dd5) [0x7fca07505dd5] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 24: (clone()+0x6d) [0x7fca063cc02d] 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-01T02:25:21.935 INFO:tasks.ceph.osd.3.mira064.stderr: 2019-10-01T02:25:22.057 INFO:tasks.rados.rados.0.mira064.stdout:106: expect (ObjNum 20 snap 0 seq_num 20) 2019-10-01T02:25:22.094 INFO:tasks.ceph.osd.5.mira064.stderr:daemon-helper: command crashed with signal 6 2019-10-01T02:25:22.297 INFO:tasks.ceph.osd.3.mira064.stderr:daemon-helper: command crashed with signal 6 2019-10-01T02:25:23.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T02:25:23.442+0000 7f109bc87700 -1 osd.0 pg_epoch: 34 pg[2.b( v 29'5 (0'0,29'5] local-lis/les=16/17 n=2 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [0,2] r=0 lpr=16 crt=29'5 lcod 19'3 mlcod 19'3 active+clean+snaptrim trimq=[7~1,b~1,d~1] ps=[2~1,6~1,8~1]] removing snap head 2019-10-01T02:25:25.784 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:25:25.776+0000 7fed41a93700 -1 osd.1 pg_epoch: 36 pg[2.7( v 32'23 (0'0,32'23] local-lis/les=16/17 n=5 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [1,0] r=0 lpr=16 crt=32'23 lcod 31'21 mlcod 31'21 active+clean+snaptrim trimq=[7~1] ps=[6~1,8~1,b~1,d~1]] removing snap head 2019-10-01T02:25:27.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2019-10-01T02:25:27.069 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2019-10-01T02:25:32.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2019-10-01T02:25:32.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2019-10-01T02:25:38.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s 2019-10-01T02:25:38.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~11s 2019-10-01T02:25:40.841 INFO:tasks.ceph.osd.1.mira063.stderr:2019-10-01T02:25:40.833+0000 7fed42a95700 -1 osd.1 pg_epoch: 36 pg[2.5( v 36'76 (0'0,36'76] local-lis/les=16/17 n=13 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) [1,4] r=0 lpr=16 crt=36'76 lcod 36'74 mlcod 36'74 active+clean+snaptrim trimq=[7~1,b~1,d~1] ps=[6~1,8~1]] removing snap head 2019-10-01T02:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~17s 2019-10-01T02:25:44.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~17s 2019-10-01T02:25:49.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~23s 2019-10-01T02:25:49.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~23s 2019-10-01T02:25:55.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~29s 2019-10-01T02:25:55.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~29s 2019-10-01T02:26:01.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~34s 2019-10-01T02:26:01.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~34s 2019-10-01T02:26:07.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~40s 2019-10-01T02:26:07.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~40s 2019-10-01T02:26:12.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~46s 2019-10-01T02:26:12.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~46s 2019-10-01T02:26:18.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~51s 2019-10-01T02:26:18.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~51s 2019-10-01T02:26:24.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~57s 2019-10-01T02:26:24.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~57s 2019-10-01T02:26:29.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~63s 2019-10-01T02:26:29.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~63s 2019-10-01T02:26:32.593 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:26:32.593 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:26:32.593 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:26:32.594 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:26:32.594 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:26:32.603 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:26:32.603 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:26:32.603 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:32.603 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:26:33.624 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:33.656 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":6,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:35.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~68s 2019-10-01T02:26:35.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~68s 2019-10-01T02:26:36.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:36.657 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:26:37.657 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:37.688 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":10,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:37.688 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:26:37.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:37.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:26:38.665 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":10,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:26:38.687 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:38.687 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:26:39.715 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":10,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:26:39.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:39.747 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:26:40.732 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":10,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:26:40.762 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:26:40.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:40.763 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:26:41.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~74s 2019-10-01T02:26:41.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~74s 2019-10-01T02:26:41.867 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:26:42.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:26:42.869 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:42.869 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:26:43.839 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:43.861 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":10,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:26:43.862 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:26:43.862 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:26:43.862 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:43.862 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:26:44.881 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":10,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:26:44.911 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:44.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:26:45.928 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":10,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:26:45.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:26:45.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:26:46.933 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":10,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:26:46.955 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:26:46.955 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T02:26:46.955 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:26:46.955 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:26:46.956 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:26:46.956 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:26:46.974 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:26:46.974 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T02:26:46.974 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T02:26:46.975 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T02:26:47.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~80s 2019-10-01T02:26:47.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~80s 2019-10-01T02:26:52.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~86s 2019-10-01T02:26:52.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~86s 2019-10-01T02:26:58.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~92s 2019-10-01T02:26:58.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~92s 2019-10-01T02:27:01.975 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T02:27:01.975 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T02:27:01.975 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:27:01.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:27:01.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:27:04.306 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~97s 2019-10-01T02:27:04.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~97s 2019-10-01T02:27:09.186 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:27:09.208 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":12,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:27:09.208 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:27:09.208 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:27:09.209 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:27:10.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~103s 2019-10-01T02:27:10.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~103s 2019-10-01T02:27:10.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":12,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:27:10.243 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:27:10.243 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:27:11.273 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":12,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:27:11.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:27:15.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~109s 2019-10-01T02:27:15.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~109s 2019-10-01T02:27:21.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~114s 2019-10-01T02:27:21.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~114s 2019-10-01T02:27:27.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2019-10-01T02:27:27.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~120s 2019-10-01T02:27:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~126s 2019-10-01T02:27:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~126s 2019-10-01T02:27:38.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~131s 2019-10-01T02:27:38.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~131s 2019-10-01T02:27:44.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~137s 2019-10-01T02:27:44.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~137s 2019-10-01T02:27:49.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~143s 2019-10-01T02:27:49.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~143s 2019-10-01T02:27:55.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~149s 2019-10-01T02:27:55.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~149s 2019-10-01T02:28:01.385 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~154s 2019-10-01T02:28:01.385 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~154s 2019-10-01T02:28:07.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~160s 2019-10-01T02:28:07.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~160s 2019-10-01T02:28:12.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~166s 2019-10-01T02:28:12.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~166s 2019-10-01T02:28:18.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~171s 2019-10-01T02:28:18.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~171s 2019-10-01T02:28:24.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~177s 2019-10-01T02:28:24.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~177s 2019-10-01T02:28:29.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~183s 2019-10-01T02:28:29.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~183s 2019-10-01T02:28:35.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~189s 2019-10-01T02:28:35.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~189s 2019-10-01T02:28:41.296 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:28:41.296 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:28:41.296 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:28:41.296 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:28:41.297 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:28:41.303 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:28:41.303 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T02:28:41.303 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T02:28:41.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T02:28:41.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~194s 2019-10-01T02:28:41.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~194s 2019-10-01T02:28:47.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~200s 2019-10-01T02:28:47.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~200s 2019-10-01T02:28:53.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~206s 2019-10-01T02:28:53.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~206s 2019-10-01T02:28:56.305 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T02:28:56.305 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T02:28:56.305 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:28:56.306 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:28:56.306 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:28:57.307 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:28:57.338 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":16,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:28:57.338 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:28:57.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:28:57.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:28:58.358 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":16,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:28:58.380 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:28:58.381 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:28:58.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~212s 2019-10-01T02:28:58.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~212s 2019-10-01T02:28:59.303 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":16,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:28:59.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:28:59.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:29:00.314 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":16,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:29:00.337 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:29:00.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:00.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:29:00.947 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T02:29:00.925+0000 7f340cd7e700 0 --2- 172.21.6.124:0/2603890785 >> v2:172.21.6.124:3300/0 conn(0x7f340813d8f0 0x7f340813fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T02:29:01.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:29:02.690 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:29:02.691 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:02.691 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:29:03.703 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:29:03.724 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":16,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:29:03.725 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:29:03.725 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:29:03.725 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:03.725 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:29:04.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~217s 2019-10-01T02:29:04.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~217s 2019-10-01T02:29:04.743 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":16,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:29:04.774 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:04.774 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:29:05.787 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":16,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:29:05.810 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:05.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:29:06.838 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":16,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:29:06.860 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:29:06.860 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T02:29:06.860 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:29:06.860 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:29:06.861 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:29:06.861 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:29:06.881 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:29:06.881 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T02:29:06.881 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T02:29:06.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T02:29:10.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~223s 2019-10-01T02:29:10.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~223s 2019-10-01T02:29:16.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~229s 2019-10-01T02:29:16.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~229s 2019-10-01T02:29:21.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~235s 2019-10-01T02:29:21.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~235s 2019-10-01T02:29:21.883 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T02:29:21.883 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T02:29:21.883 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:29:21.883 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:21.884 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:29:27.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~240s 2019-10-01T02:29:27.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~240s 2019-10-01T02:29:27.543 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:29:27.576 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":20,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:29:27.576 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:29:27.576 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:27.577 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:29:28.585 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":20,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:29:28.608 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:29:28.608 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:29:29.574 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":20,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:29:29.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:29:33.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~246s 2019-10-01T02:29:33.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~246s 2019-10-01T02:29:38.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~252s 2019-10-01T02:29:38.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~252s 2019-10-01T02:29:44.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~258s 2019-10-01T02:29:44.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~258s 2019-10-01T02:29:50.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~263s 2019-10-01T02:29:50.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~263s 2019-10-01T02:29:56.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~269s 2019-10-01T02:29:56.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~269s 2019-10-01T02:30:01.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~275s 2019-10-01T02:30:01.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~275s 2019-10-01T02:30:07.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~280s 2019-10-01T02:30:07.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~280s 2019-10-01T02:30:13.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~286s 2019-10-01T02:30:13.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~286s 2019-10-01T02:30:18.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~292s 2019-10-01T02:30:18.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~292s 2019-10-01T02:30:24.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~298s 2019-10-01T02:30:24.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~298s 2019-10-01T02:30:30.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~303s 2019-10-01T02:30:30.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~303s 2019-10-01T02:30:30.284 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-10-01T02:30:30.284 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa/tasks/daemonwatchdog.py", line 38, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa/tasks/daemonwatchdog.py", line 114, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-09-30-1336/qa/tasks/daemonwatchdog.py", line 54, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 247, in __getattr__ raise AttributeError(name) AttributeError: mounts 2019-10-01T02:30:59.598 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:30:59.598 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:30:59.598 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:30:59.598 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:30:59.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:30:59.608 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:30:59.608 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T02:30:59.609 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T02:30:59.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T02:31:14.609 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T02:31:14.610 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T02:31:14.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:31:14.610 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:14.610 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:15.627 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:15.649 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:15.649 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:31:15.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:15.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:31:16.649 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":28,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:16.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:16.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:31:17.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":28,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:17.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:17.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:31:18.712 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":28,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:18.732 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:31:18.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:18.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:31:19.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:31:20.853 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:31:20.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:20.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:21.871 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:21.891 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:21.892 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:31:21.892 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:31:21.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:21.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:31:22.890 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":28,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:22.921 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:22.922 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:31:23.885 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":28,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:23.916 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:23.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:31:24.900 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":28,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:24.922 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:31:24.922 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:31:24.922 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:31:24.922 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:31:24.923 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:31:24.923 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:31:24.946 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:31:24.946 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:31:24.946 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:24.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:25.948 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:25.972 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:28.974 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:28.974 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:29.963 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:29.985 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:32.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:32.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:33.990 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:34.012 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:37.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:37.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:31:40.252 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:40.276 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:31:40.276 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:31:40.277 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:40.277 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:31:41.291 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":30,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:31:41.314 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:31:41.314 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:31:42.303 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":30,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:31:42.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:33:12.339 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:33:12.339 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:33:12.339 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:33:12.339 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:33:12.340 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:33:12.346 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:33:12.346 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:33:12.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:12.347 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:13.363 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:13.389 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:16.390 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:16.390 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:17.377 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:17.408 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:20.409 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:20.409 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:21.404 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:21.436 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:24.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:24.437 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:25.504 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:25.526 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":30,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:28.527 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:28.527 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:29.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:29.543 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:29.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:33:29.543 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:29.544 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:33:30.531 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":34,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:33:30.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:30.558 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:33:31.539 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":34,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:33:31.571 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:31.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:33:32.574 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":34,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:33:32.596 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:33:32.597 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:32.597 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:33:33.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:33:34.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:33:34.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:34.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:35.681 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:35.712 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:35.712 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:33:35.712 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:33:35.712 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:35.713 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:33:36.718 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":34,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:33:36.748 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:36.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:33:37.782 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":34,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:33:37.809 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:37.809 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:33:38.895 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":34,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:33:38.927 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T02:33:38.928 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:33:38.928 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T02:33:38.928 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:33:38.928 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T02:33:38.928 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:33:38.940 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T02:33:38.941 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:33:38.941 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:38.941 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:40.018 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:40.048 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:43.050 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:43.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:44.052 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:44.083 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:47.084 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:47.084 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:48.093 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:48.111 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":34,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:51.112 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:51.113 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:33:54.409 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:54.438 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":38,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:33:54.438 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:33:54.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:54.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:33:55.451 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":38,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:33:55.476 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:33:55.476 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:33:56.488 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":38,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:33:56.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:35:26.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:35:26.512 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T02:35:26.513 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T02:35:26.513 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:26.513 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T02:35:26.519 INFO:tasks.ceph.mon.c:Started 2019-10-01T02:35:26.519 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:35:26.519 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:26.520 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:27.478 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:27.501 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":38,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:30.502 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:30.502 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:35.051 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:35.084 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":42,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:38.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:38.086 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:39.098 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:39.129 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:39.129 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:35:39.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:39.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:35:40.139 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":44,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:35:40.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:40.161 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:35:41.167 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":44,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:35:41.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:41.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:35:42.204 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":44,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:35:42.226 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:35:42.227 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:42.227 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:35:43.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:35:44.338 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:35:44.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:44.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:45.327 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:45.359 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:45.359 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:35:45.359 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:35:45.360 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:45.360 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:35:46.378 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":44,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:35:46.410 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:46.411 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:35:47.394 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":44,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:35:47.424 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:47.424 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:35:48.420 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":44,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:35:48.443 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:35:48.443 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:35:48.443 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:35:48.443 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:35:48.443 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:35:48.444 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:35:48.462 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:35:48.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:35:48.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:48.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:49.507 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:49.537 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:52.538 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:52.538 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:53.496 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:53.519 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:56.520 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:35:56.520 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:35:57.518 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:35:57.541 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":44,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:36:00.542 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:36:00.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:36:03.491 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:36:03.514 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:36:03.514 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:36:03.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:36:03.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:36:04.534 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":48,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:36:04.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:36:04.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:36:05.560 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":48,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:36:05.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:37:35.584 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:37:35.584 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:37:35.584 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:37:35.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:35.585 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:37:35.593 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:37:35.593 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:37:35.594 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:35.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:36.597 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:36.627 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:39.628 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:39.629 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:40.638 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:40.817 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":48,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:43.818 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:43.818 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:44.835 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:44.858 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:44.858 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:37:44.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:44.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:37:45.958 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":50,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:37:45.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:45.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:37:46.987 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":50,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:37:47.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:47.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:37:48.019 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":50,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:37:48.048 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:37:48.049 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:48.049 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:37:49.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:37:50.125 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:37:50.126 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:50.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:51.125 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:51.156 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:51.156 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:37:51.156 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:37:51.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:51.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:37:52.177 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":50,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:37:52.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:52.201 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:37:53.202 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":50,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:37:53.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:53.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:37:54.249 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":50,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:37:54.272 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T02:37:54.272 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:37:54.272 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T02:37:54.272 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:37:54.273 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T02:37:54.273 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:37:54.286 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T02:37:54.286 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:37:54.286 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:54.286 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:55.346 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:55.369 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:58.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:37:58.370 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:37:59.359 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:37:59.393 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:38:02.394 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:38:02.394 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:38:03.398 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:38:03.621 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":50,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:38:06.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:38:06.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:38:10.677 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:38:10.699 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":54,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:38:10.699 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:38:10.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:38:10.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:38:11.664 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":54,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:38:11.696 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:38:11.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:38:12.697 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":54,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:38:12.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:39:42.720 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:39:42.720 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T02:39:42.720 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T02:39:42.720 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:42.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T02:39:42.727 INFO:tasks.ceph.mon.c:Started 2019-10-01T02:39:42.727 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:39:42.727 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:42.727 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:39:43.717 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:43.746 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":54,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:46.748 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:46.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:39:52.341 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:52.372 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":58,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:55.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:55.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:39:56.379 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:56.408 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:39:56.408 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:39:56.408 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:56.408 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:39:57.384 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":60,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:39:57.406 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:57.407 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:39:58.404 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":60,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:39:58.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:58.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:39:59.402 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":60,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:39:59.419 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:39:59.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:39:59.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:40:00.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:40:01.534 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:40:01.534 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:01.535 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:40:02.516 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:02.549 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:02.549 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:40:02.549 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:40:02.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:02.550 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:40:03.536 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":60,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:40:03.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:03.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:40:04.556 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":60,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:40:04.589 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:04.589 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:40:05.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":60,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:40:05.632 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:40:05.632 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:40:05.633 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:40:05.633 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:40:05.633 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:40:05.633 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:40:05.652 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:40:05.652 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:40:05.652 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:05.652 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:40:06.616 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:06.637 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:09.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:09.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:40:10.637 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:10.667 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:13.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:13.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:40:14.678 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:14.700 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":60,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:17.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:17.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:40:21.732 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:21.763 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:40:21.764 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:40:21.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:21.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:40:22.770 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":62,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:40:22.802 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:40:22.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:40:23.803 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":62,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:40:23.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:41:53.835 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:41:53.835 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:41:53.836 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:41:53.836 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:41:53.836 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:41:53.842 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:41:53.842 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:41:53.842 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:41:53.842 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:41:54.860 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:41:54.880 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:41:57.881 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:41:57.881 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:41:58.853 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:41:58.884 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":62,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:01.886 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:01.886 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:07.455 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:07.477 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:07.477 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:42:07.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:07.477 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:42:08.422 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":70,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:42:08.445 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:08.446 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:42:09.487 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":70,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:42:09.517 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:09.517 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:42:10.484 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":70,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:42:10.506 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:42:10.507 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:10.507 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:42:11.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:42:12.622 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:42:12.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:12.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:13.644 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:13.675 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:13.675 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:42:13.676 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:42:13.676 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:13.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:42:14.677 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":70,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:42:14.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:14.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:42:15.693 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":70,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:42:15.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:15.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:42:16.689 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":70,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:42:16.715 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:42:16.715 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:42:16.715 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:42:16.716 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:42:16.716 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:42:16.716 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:42:16.732 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:42:16.732 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:42:16.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:16.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:17.776 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:17.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:20.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:20.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:21.790 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:21.812 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:24.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:24.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:25.812 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:25.844 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":70,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:28.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:28.846 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:42:32.885 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:32.915 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:42:32.915 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:42:32.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:32.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:42:33.929 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":74,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:42:33.951 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:42:33.951 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:42:34.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":74,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:42:34.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:44:04.893 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:44:04.894 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:44:04.894 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:44:04.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:04.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:44:04.902 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:44:04.902 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:44:04.903 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:04.903 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:05.855 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:05.884 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:08.885 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:08.886 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:09.879 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:09.901 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":74,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:12.902 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:12.902 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:17.405 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:17.427 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:17.427 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:44:17.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:17.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:44:18.439 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":82,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:18.473 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:18.473 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:44:19.461 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":82,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:44:19.483 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:19.483 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:44:20.484 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":82,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:20.506 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:44:20.506 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:20.506 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:44:21.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:44:22.645 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:44:22.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:22.646 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:23.641 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:23.671 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:23.671 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:44:23.672 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:44:23.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:23.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:44:24.739 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":82,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:24.770 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:24.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:44:25.786 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":82,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:44:25.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:25.818 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:44:26.824 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":82,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:26.844 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:44:26.844 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:44:26.845 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:44:26.845 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:44:26.845 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:44:26.845 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:44:26.867 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:44:26.867 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:44:26.867 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:26.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:27.862 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:27.883 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:30.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:30.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:31.874 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:31.903 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":82,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:34.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:34.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:44:41.956 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:41.980 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:44:41.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:44:41.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:41.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:44:42.954 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":84,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:42.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:44:42.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:44:43.945 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":84,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:44:43.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:46:13.973 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:46:13.973 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:46:13.973 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:46:13.974 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:46:13.974 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:46:13.979 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:46:13.980 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:46:13.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:13.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:14.984 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:15.014 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:18.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:18.017 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:19.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:19.084 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":84,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:22.085 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:22.085 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:26.720 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":88,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:26.739 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":88,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:29.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:29.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:30.725 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":88,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:30.755 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":88,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:33.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:33.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:34.714 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:34.741 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:34.742 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:46:34.742 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:34.742 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:46:35.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":92,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:35.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:35.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:46:36.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":92,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:36.826 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:36.826 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:46:37.819 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":92,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:37.840 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:46:37.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:37.841 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:46:38.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:46:39.994 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:46:39.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:39.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:41.009 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:41.038 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:41.038 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:46:41.039 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:46:41.039 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:41.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:46:42.044 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":92,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:42.074 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:42.074 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:46:43.049 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":92,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:43.073 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:43.074 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:46:44.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":92,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:46:44.095 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:46:44.095 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:46:44.095 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:46:44.095 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:46:44.096 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:46:44.096 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:46:44.114 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:46:44.114 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:46:44.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:44.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:45.117 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:45.139 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:48.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:48.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:49.140 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:49.159 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:52.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:52.161 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:53.158 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:53.188 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":92,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:56.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:56.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:46:59.479 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":94,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:59.509 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":94,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:46:59.509 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:46:59.509 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:46:59.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:47:00.487 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":94,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:47:00.509 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:47:00.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:47:01.525 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":94,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:47:01.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:48:31.548 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:48:31.548 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:48:31.548 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:48:31.548 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:48:31.549 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:48:31.559 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:48:31.559 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:48:31.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:31.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:48:32.563 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":94,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:32.593 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":94,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:35.594 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:35.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:48:41.036 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":98,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:41.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":98,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:44.060 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:44.060 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:48:47.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":102,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:47.088 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":102,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:50.090 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:50.090 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:48:53.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":106,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:53.105 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":106,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:56.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:56.106 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:48:57.950 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:57.982 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:48:57.982 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:48:57.983 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:57.983 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:48:58.976 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":110,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:48:58.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:48:58.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:49:00.004 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":110,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:00.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:00.037 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:49:01.028 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":110,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:01.051 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:49:01.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:01.051 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:49:02.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:49:03.170 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:49:03.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:03.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:49:04.166 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:04.198 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:04.198 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:49:04.199 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:49:04.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:04.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:49:05.169 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":110,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:05.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:05.201 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:49:06.205 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":110,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:06.231 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:06.232 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:49:07.263 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":110,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:07.289 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:49:07.289 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:49:07.289 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:49:07.289 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:49:07.290 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:49:07.290 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:49:07.315 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:49:07.315 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:49:07.315 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:07.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:49:08.392 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:08.417 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:11.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:11.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:49:12.425 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:12.455 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:15.456 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:15.456 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:49:16.466 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:16.488 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":110,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:19.490 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:19.490 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:49:23.549 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:23.579 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:49:23.579 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:49:23.580 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:23.580 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:49:24.599 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":114,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:24.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:49:24.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:49:25.608 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":114,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:49:25.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:50:55.638 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:50:55.639 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:50:55.639 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:50:55.639 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:50:55.639 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:50:55.725 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:50:55.725 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:50:55.726 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:50:55.726 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:50:56.758 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:50:56.795 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:50:59.798 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:50:59.798 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:00.814 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:00.836 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":114,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:03.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:03.837 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:04.805 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:04.837 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:04.837 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:51:04.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:04.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:51:05.881 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":116,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:05.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:05.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:51:06.940 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":116,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:06.972 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:06.972 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:51:07.919 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":116,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:07.938 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:51:07.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:07.938 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:51:09.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:51:10.042 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:51:10.043 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:10.043 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:11.032 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:11.055 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:11.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:51:11.055 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:51:11.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:11.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:51:12.073 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":116,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:12.096 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:12.096 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:51:13.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":116,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:13.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:13.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:51:14.140 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":116,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:14.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T02:51:14.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:51:14.162 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T02:51:14.162 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:51:14.162 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T02:51:14.163 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:51:14.187 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T02:51:14.187 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:51:14.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:14.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:15.189 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:15.221 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:18.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:18.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:19.213 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:19.236 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:22.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:22.237 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:51:29.638 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:29.670 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:51:29.670 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:51:29.671 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:29.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:51:30.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":118,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:30.682 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:51:30.682 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:51:31.681 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":118,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:51:31.703 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:53:01.703 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T02:53:01.704 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T02:53:01.704 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T02:53:01.704 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T02:53:01.704 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T02:53:01.710 INFO:tasks.ceph.mon.b:Started 2019-10-01T02:53:01.710 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:53:01.711 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:01.711 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:02.713 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:02.734 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:05.735 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:05.735 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:06.713 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:06.744 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:09.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:09.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:10.746 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:10.778 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:13.779 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:13.779 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:14.793 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:14.823 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":118,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:17.824 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:17.824 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:18.811 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:18.843 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:18.843 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:53:18.843 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:18.843 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:53:19.820 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":122,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:19.843 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:19.843 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:53:20.801 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":122,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:20.823 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:20.823 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:53:21.777 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":122,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:21.798 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:53:21.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:21.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:53:22.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:53:23.923 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:53:23.923 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:23.923 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:24.935 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:24.958 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:24.958 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:53:24.958 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:53:24.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:24.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:53:25.959 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":122,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:25.990 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:25.990 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:53:26.969 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":122,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:26.993 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:26.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:53:27.947 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":122,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:27.969 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T02:53:27.970 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:53:27.970 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T02:53:27.970 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:53:27.970 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T02:53:27.970 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:53:27.990 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T02:53:27.990 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:53:27.990 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:27.990 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:29.068 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:29.090 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:32.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:32.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:33.082 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:33.105 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:36.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:36.106 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:53:42.082 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:42.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":126,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:53:42.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:53:42.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:42.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:53:43.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":126,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:43.138 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:53:43.138 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:53:44.131 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":126,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:53:44.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:55:14.154 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T02:55:14.155 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T02:55:14.155 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T02:55:14.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:14.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T02:55:14.161 INFO:tasks.ceph.mon.c:Started 2019-10-01T02:55:14.161 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:55:14.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:14.162 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:15.153 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":126,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:15.176 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":126,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:18.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:18.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:25.398 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:25.429 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:25.429 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:55:25.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:25.430 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:55:26.439 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":132,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:26.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:26.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:55:27.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":132,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:27.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:27.725 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:55:28.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":132,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:28.812 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:55:28.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:28.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:55:29.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:55:30.891 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:55:30.891 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:30.891 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:31.881 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:31.904 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:31.904 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:55:31.904 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:55:31.905 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:31.905 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:55:32.926 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":132,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:32.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:32.959 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:55:33.953 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":132,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:33.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:33.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:55:34.977 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":132,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:35.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:55:35.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:55:35.000 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:55:35.000 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:55:35.000 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:55:35.001 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:55:35.019 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:55:35.020 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:55:35.020 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:35.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:36.052 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:36.075 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:39.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:39.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:40.087 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:40.109 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:43.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:43.110 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:44.102 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:44.134 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":132,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:47.135 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:47.135 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:55:51.200 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:51.222 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:55:51.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:55:51.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:51.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:55:52.232 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":136,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:55:52.254 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:55:52.254 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:55:53.267 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":136,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:55:53.289 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:57:23.290 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:57:23.290 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:57:23.290 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:57:23.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:23.291 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:57:23.297 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:57:23.297 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:57:23.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:23.297 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:24.195 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:24.217 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:27.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:27.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:28.220 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:28.243 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":136,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:31.244 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:31.244 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:32.277 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:32.299 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:32.299 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:57:32.300 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:32.300 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:57:33.331 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":138,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:57:33.360 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:33.361 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:57:34.494 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":138,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:57:34.521 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:34.521 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:57:35.519 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":138,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:57:35.542 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:57:35.542 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:35.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:57:36.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:57:37.642 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:57:37.771 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:37.771 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:38.717 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:38.739 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:38.739 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:57:38.740 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:57:38.740 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:38.740 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:57:39.750 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":138,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:57:39.780 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:39.780 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:57:40.790 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":138,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:57:40.811 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:40.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:57:41.825 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":138,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:57:41.847 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:57:41.848 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:57:41.848 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:57:41.848 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:57:41.849 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:57:41.849 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:57:41.868 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:57:41.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:57:41.868 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:41.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:42.913 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:42.943 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:45.944 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:45.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:46.926 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:46.949 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:49.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:49.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:57:56.984 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:57.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:57:57.013 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T02:57:57.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:57.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:57:57.999 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":142,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:57:58.029 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:57:58.029 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:57:59.044 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":142,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T02:57:59.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T02:59:29.073 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:59:29.073 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T02:59:29.073 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T02:59:29.074 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:29.074 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T02:59:29.127 INFO:tasks.ceph.mon.a:Started 2019-10-01T02:59:29.128 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:59:29.128 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:29.128 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:34.896 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:34.928 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":146,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:37.929 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:37.929 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:40.491 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:40.513 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:40.513 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:59:40.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:40.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:59:41.544 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":154,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:41.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:41.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:59:42.585 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":154,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:42.613 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:42.613 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:59:43.620 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":154,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:43.650 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T02:59:43.650 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:43.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T02:59:44.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T02:59:45.780 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T02:59:45.781 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:45.781 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:46.802 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:46.824 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:46.824 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T02:59:46.824 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T02:59:46.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:46.825 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T02:59:47.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":154,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:47.818 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:47.819 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T02:59:48.777 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":154,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:48.809 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:48.809 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T02:59:49.769 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":154,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T02:59:49.790 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T02:59:49.790 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T02:59:49.791 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T02:59:49.791 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T02:59:49.791 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T02:59:49.791 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T02:59:49.810 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T02:59:49.810 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T02:59:49.811 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:49.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:50.846 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:50.879 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:53.880 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:53.880 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:54.881 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:54.903 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:57.905 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T02:59:57.905 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T02:59:58.884 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T02:59:58.902 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:00:01.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:00:01.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:00:05.965 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":158,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:00:05.993 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":158,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:00:05.993 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:00:05.994 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:00:05.994 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:00:07.015 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":158,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:00:07.042 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:00:07.042 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:00:08.055 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":158,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:00:08.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:01:38.079 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:01:38.079 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:01:38.079 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:01:38.079 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:38.079 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:01:38.088 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:01:38.088 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:01:38.089 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:38.089 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:01:43.857 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:43.889 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:46.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:46.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:01:49.485 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:49.517 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:49.517 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:01:49.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:49.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:01:50.509 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":170,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:50.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:50.541 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:01:51.525 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":170,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:51.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:51.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:01:52.527 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":170,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:52.549 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:01:52.549 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:52.549 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:01:53.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:01:54.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:01:54.630 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:54.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:01:55.647 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:55.677 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:55.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:01:55.678 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:01:55.678 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:55.678 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:01:56.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":170,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:56.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:56.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:01:57.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":170,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:57.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:57.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:01:58.745 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":170,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:01:58.776 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:01:58.776 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:01:58.776 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:01:58.776 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:01:58.777 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:01:58.777 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:01:58.806 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:01:58.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:01:58.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:01:58.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:01:59.861 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:01:59.894 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:02.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:02:02.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:02:03.907 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:03.928 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:06.929 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:02:06.930 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:02:07.916 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:07.940 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":170,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:10.941 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:02:10.941 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:02:14.982 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:15.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:02:15.013 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:02:15.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:02:15.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:02:16.011 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":172,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:02:16.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:02:16.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:02:17.059 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":172,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:02:17.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:03:47.090 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:03:47.091 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:03:47.091 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:03:47.091 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:03:47.091 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:03:47.097 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:03:47.097 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:03:47.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:03:47.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:03:48.105 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:03:48.128 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:03:51.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:03:51.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:03:55.277 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":176,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:03:55.299 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":176,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:03:58.299 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:03:58.300 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:04:00.784 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:00.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":180,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:00.806 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:04:00.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:00.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:04:01.765 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":180,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:04:01.790 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:01.790 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:04:02.718 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":180,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:04:02.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:02.746 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:04:03.704 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":180,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:04:03.727 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:04:03.727 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:03.727 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:04:04.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:04:05.829 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:04:05.830 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:05.830 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:04:06.808 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":180,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:06.830 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":180,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:06.830 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:04:06.831 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:04:06.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:06.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:04:07.811 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":180,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:04:07.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:07.842 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:04:08.731 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":180,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:04:08.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:08.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:04:09.730 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":180,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:04:09.751 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:04:09.752 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'b'] 2019-10-01T03:04:09.752 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:04:09.752 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:04:09.752 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:04:09.752 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:04:09.778 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:04:09.779 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T03:04:09.779 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T03:04:09.779 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:04:09.779 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:04:09.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:04:24.780 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T03:04:24.781 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T03:04:24.781 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:04:24.782 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:04:24.782 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:04:24.782 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:24.783 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:04:30.420 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":184,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:30.449 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":184,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:04:30.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:04:30.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:30.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:04:31.450 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":184,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:04:31.479 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:04:31.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:04:32.469 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":184,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:04:32.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:06:02.499 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:06:02.499 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:06:02.499 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:06:02.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:02.500 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:06:02.507 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:06:02.507 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T03:06:02.507 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T03:06:02.507 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:06:02.508 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:06:02.508 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:06:17.509 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T03:06:17.509 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T03:06:17.509 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:06:17.510 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:06:17.510 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:06:17.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:17.510 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:23.154 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:23.184 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:23.185 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:06:23.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:23.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:06:24.173 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":190,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:06:24.203 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:24.203 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:06:31.403 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":190,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:31.425 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:31.425 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:06:32.427 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":190,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:32.450 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:06:32.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:32.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:06:33.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:06:34.558 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:06:34.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:34.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:35.539 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:35.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:35.560 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:06:35.560 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:06:35.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:35.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:06:36.556 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":190,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:06:36.576 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:36.577 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:06:37.582 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":190,"quorum":[0,1,2],"quorum_age":14,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:37.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:37.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:06:38.575 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":190,"quorum":[0,1,2],"quorum_age":15,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:38.597 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:06:38.597 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:06:38.597 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:06:38.597 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:06:38.598 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:06:38.598 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:06:38.657 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:06:38.658 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:06:38.658 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:38.658 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:39.666 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:39.688 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:42.689 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:42.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:43.670 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:43.692 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:46.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:46.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:47.680 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":24,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:47.709 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":190,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":24,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:50.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:50.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:06:54.742 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":194,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:54.772 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":194,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:06:54.773 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:06:54.773 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:54.773 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:06:55.772 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":194,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:55.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:06:55.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:06:56.816 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":194,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:06:56.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:08:26.847 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:08:26.847 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:08:26.848 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:08:26.848 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:26.848 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:08:26.854 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:08:26.854 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:08:26.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:26.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:27.845 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":194,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:27.875 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":194,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:30.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:30.877 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:39.889 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:39.918 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:39.919 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:08:39.919 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:39.919 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:08:40.908 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":206,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:40.939 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:40.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:08:41.945 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":206,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:41.970 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:41.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:08:42.981 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":206,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:43.009 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:08:43.009 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:43.009 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:08:44.123 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:08:45.123 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:08:45.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:45.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:46.114 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:46.136 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:46.137 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:08:46.137 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:08:46.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:46.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:08:47.149 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":206,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:47.179 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:47.179 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:08:48.142 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":206,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:48.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:48.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:08:49.174 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":206,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:08:49.196 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:08:49.196 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:08:49.196 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:08:49.197 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:08:49.197 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:08:49.197 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:08:49.214 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:08:49.214 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:08:49.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:49.215 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:50.286 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:50.309 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:53.309 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:53.310 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:54.310 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:54.332 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:57.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:08:57.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:08:58.349 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:08:58.372 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":206,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:09:01.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:09:01.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:09:04.426 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":210,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:09:04.451 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":210,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:09:04.451 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:09:04.451 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:09:04.451 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:09:05.443 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":210,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:09:05.473 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:09:05.474 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:09:06.491 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":210,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:09:06.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:10:36.514 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:10:36.515 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:10:36.515 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:10:36.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:36.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:10:36.522 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:10:36.522 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:10:36.522 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:36.522 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:10:37.612 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":210,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:37.633 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":210,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:40.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:40.635 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:10:46.172 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:46.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":218,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:46.195 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:10:46.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:46.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:10:47.196 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":218,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:47.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:47.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:10:48.197 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":218,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:48.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:48.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:10:49.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":218,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:49.233 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:10:49.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:49.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:10:50.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:10:51.340 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:10:51.341 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:51.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:10:52.406 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:52.427 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":218,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:10:52.427 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:10:52.427 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:10:52.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:52.427 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:10:53.429 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":218,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:53.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:53.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:10:54.472 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":218,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:54.495 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:10:54.495 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:10:55.499 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":218,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:10:55.529 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:10:55.529 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T03:10:55.529 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:10:55.529 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:10:55.529 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:10:55.530 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:10:55.550 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:10:55.550 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:10:55.551 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:10:55.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:11:10.551 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:11:10.552 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:11:10.552 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:11:10.552 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:11:10.552 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:11:16.211 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":222,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:11:16.233 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":222,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:11:16.234 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:11:16.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:11:16.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:11:17.235 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":222,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:11:17.265 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:11:17.265 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:11:18.273 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":222,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:11:18.296 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:12:48.296 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:12:48.297 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:12:48.297 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:12:48.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:12:48.297 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:12:48.305 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:12:48.305 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:12:48.305 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:12:48.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:13:03.306 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:13:03.307 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:13:03.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:13:03.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:03.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:08.971 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":226,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:08.996 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":226,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:11.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:11.997 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:13.044 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:13.067 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:13.067 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:13:13.067 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:13.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:13:14.152 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":230,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:14.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:14.175 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:13:15.138 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":230,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:15.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:15.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:13:16.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":230,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:16.191 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:13:16.191 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:16.191 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:13:17.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:13:18.326 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:13:18.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:18.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:19.419 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:19.445 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:19.446 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:13:19.446 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:13:19.446 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:19.446 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:13:20.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":230,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:20.644 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:20.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:13:21.651 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":230,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:21.676 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:21.676 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:13:22.685 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":230,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:22.703 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:13:22.703 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:13:22.704 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:13:22.704 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:13:22.704 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:13:22.704 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:13:22.722 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:13:22.723 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:13:22.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:22.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:23.774 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:23.804 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:26.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:26.805 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:27.808 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:27.831 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:30.832 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:30.832 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:31.826 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:31.856 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":230,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:34.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:34.857 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:13:37.689 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":234,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:37.712 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":234,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:13:37.712 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:13:37.712 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:37.712 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:13:38.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":234,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:38.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:13:38.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:13:39.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":234,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:13:39.796 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:14:19.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:19.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:20.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:20.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:21.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:21.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:22.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:22.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:23.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:23.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:24.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:24.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:25.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:25.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:26.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:26.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:27.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:27.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:28.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:28.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:29.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:29.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:30.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:30.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:31.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:31.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:32.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:32.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:33.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:33.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:34.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:34.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:35.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:35.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:36.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:36.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:37.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:37.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:38.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:38.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:39.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:39.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:40.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:40.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:41.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:41.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:42.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:42.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:43.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:43.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:44.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:44.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:45.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:45.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:46.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:46.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:47.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:47.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:48.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:48.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:49.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:49.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:50.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:50.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:51.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:51.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:52.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:52.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:53.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:53.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:54.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:54.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:55.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:55.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:56.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:56.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:57.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:57.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:58.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:58.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:14:59.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:14:59.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:00.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:00.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:01.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:01.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:02.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:02.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:03.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:03.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:04.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:04.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:05.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:05.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:06.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:06.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:07.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:07.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:08.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:08.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:09.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:09.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:09.797 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:15:09.797 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:15:09.798 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:15:09.798 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:09.798 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:15:09.813 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:15:09.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:15:09.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:09.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:10.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:10.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:10.808 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":234,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:10.837 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":234,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:11.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:11.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:12.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:12.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:13.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:13.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:13.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:13.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:14.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:14.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:15.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:15.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:16.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:16.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:17.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:17.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:18.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:18.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:18.793 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":238,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:18.831 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":238,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:19.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:19.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:20.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:20.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:21.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:21.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:21.832 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:21.833 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:22.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:22.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:22.829 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:22.849 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:22.850 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:15:22.850 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:22.850 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:15:23.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:23.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:23.823 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":240,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:23.855 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:23.855 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:15:24.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:24.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:24.851 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":240,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:24.882 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:24.882 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:15:25.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:25.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:25.877 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":240,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:25.899 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:15:25.899 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:25.899 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:15:26.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:26.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:26.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:15:27.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:27.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:27.997 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:15:27.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:27.997 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:28.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:28.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:28.993 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:29.024 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:29.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:15:29.024 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:15:29.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:29.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:15:29.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:29.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:30.007 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":240,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:30.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:30.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:15:30.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:30.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:31.035 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":240,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:31.058 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:31.058 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:15:31.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:31.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:32.068 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":240,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:32.096 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:15:32.097 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:15:32.097 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:15:32.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:15:32.097 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:15:32.097 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:15:32.132 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:15:32.132 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:15:32.132 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:32.132 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:32.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:32.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:33.197 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:33.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:33.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:33.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:34.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:34.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:35.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:35.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:36.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:36.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:36.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:36.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:37.209 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:37.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:37.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:37.242 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:38.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:38.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:39.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:39.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:40.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:40.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:40.243 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:40.243 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:41.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:41.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:41.253 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:41.283 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":240,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:42.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:42.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:43.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:43.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:44.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:44.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:44.285 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:44.285 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:15:45.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:45.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:46.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:46.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:47.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:47.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:48.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:48.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:48.316 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:48.347 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:15:48.348 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:15:48.348 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:48.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:15:49.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:49.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:49.321 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":244,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:49.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:15:49.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:15:50.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:50.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:50.334 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":244,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:15:50.364 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:15:51.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:51.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:52.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:52.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:53.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:53.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:54.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:54.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:55.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:55.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:56.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:56.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:57.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:57.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:58.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:58.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:15:59.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:15:59.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:00.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:00.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:01.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:01.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:01.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:01.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:03.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:03.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:04.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:04.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:05.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:05.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:06.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:06.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:07.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:07.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:08.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:08.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:09.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:09.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:10.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:10.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:10.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:10.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:11.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:11.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:12.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:12.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:13.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:13.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:14.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:14.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:15.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:15.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:16.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:16.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:18.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:18.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:18.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:18.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:20.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:20.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:21.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:21.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:22.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:22.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:23.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:23.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:24.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:24.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:25.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:25.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:25.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:25.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:27.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:27.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:28.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:28.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:29.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:29.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:30.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:30.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:31.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:31.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:32.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:32.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:33.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:33.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:34.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:34.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:35.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:35.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:36.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:36.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:37.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:37.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:38.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:38.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:39.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:39.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:40.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:40.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:41.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:41.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:42.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:42.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:43.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:43.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:44.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:44.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:45.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:45.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:46.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:46.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:47.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:47.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:48.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:48.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:49.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:49.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:50.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:50.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:51.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:51.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:52.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:52.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:53.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:53.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:54.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:54.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:55.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:55.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:56.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:56.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:57.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:57.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:58.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:58.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:16:59.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:16:59.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:00.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:00.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:01.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:01.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:02.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:02.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:03.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:03.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:04.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:04.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:05.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:05.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:06.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:06.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:07.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:07.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:08.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:08.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:09.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:09.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:10.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:10.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:11.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:11.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:12.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:12.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:13.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:13.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:14.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:14.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:15.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:15.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:16.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:16.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:17.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:17.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:18.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:18.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:19.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:19.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:20.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:20.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:20.365 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:17:20.366 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:17:20.366 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:17:20.366 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:20.366 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:17:20.372 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:17:20.372 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:17:20.372 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:20.372 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:17:21.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:21.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:21.343 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:21.375 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:22.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:22.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:23.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:23.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:24.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:24.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:24.376 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:24.376 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:17:25.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:25.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:25.374 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:25.392 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":244,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:26.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:26.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:27.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:27.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:28.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:28.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:28.393 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:28.393 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:17:29.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:29.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:29.436 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":246,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:29.458 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":246,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:29.459 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:17:29.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:29.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:17:30.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:30.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:30.534 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":246,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:30.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:30.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:17:31.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:31.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:31.566 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":246,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:31.602 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:31.602 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:17:32.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:32.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:32.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":246,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:32.672 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:17:32.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:32.673 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:17:33.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:33.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:33.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:17:34.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:34.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:34.764 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:17:34.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:34.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:17:35.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:35.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:35.746 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":246,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:35.767 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":246,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:17:35.767 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:17:35.767 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:17:35.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:35.772 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:17:36.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:36.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:36.763 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":246,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:36.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:36.794 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:17:37.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:37.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:37.791 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":246,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:37.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:37.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:17:38.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:38.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:38.806 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":246,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:17:38.836 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:17:38.836 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T03:17:38.836 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:17:38.836 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:17:38.837 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:17:38.837 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:17:38.865 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:17:38.865 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T03:17:38.866 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T03:17:38.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:17:39.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:39.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:40.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:40.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:41.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:41.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:42.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:42.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:43.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:43.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:44.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:44.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:45.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:45.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:46.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:46.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:47.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:47.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:48.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:48.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:49.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:49.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:50.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:50.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:51.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:51.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:52.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:52.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:53.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:53.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:53.866 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T03:17:53.867 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T03:17:53.867 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:17:53.867 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:17:53.867 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:17:54.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:54.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:55.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:55.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:56.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:56.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:57.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:57.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:58.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:58.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:17:59.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:17:59.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:00.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:00.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:01.075 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":248,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:18:01.097 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":248,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:18:01.098 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:18:01.098 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:18:01.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:18:01.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:01.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:02.105 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":248,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:18:02.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:18:02.138 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:18:02.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:02.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:03.129 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":248,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:18:03.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:18:03.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:03.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:04.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:04.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:05.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:05.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:06.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:06.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:07.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:07.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:08.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:08.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:09.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:09.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:10.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:10.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:11.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:11.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:12.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:12.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:13.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:13.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:14.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:14.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:15.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:15.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:16.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:16.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:17.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:17.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:18.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:18.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:19.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:19.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:20.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:20.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:21.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:21.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:22.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:22.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:23.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:23.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:24.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:24.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:25.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:25.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:26.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:26.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:27.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:27.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:28.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:28.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:29.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:29.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:30.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:30.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:31.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:31.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:32.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:32.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:33.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:33.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:34.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:34.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:35.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:35.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:36.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:36.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:37.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:37.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:38.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:38.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:39.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:39.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:40.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:40.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:41.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:41.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:42.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:42.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:43.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:43.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:44.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:44.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:45.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:45.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:46.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:46.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:47.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:47.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:48.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:48.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:49.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:49.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:50.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:50.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:51.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:51.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:52.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:52.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:53.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:53.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:54.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:54.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:55.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:55.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:56.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:56.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:57.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:57.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:58.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:58.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:18:59.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:18:59.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:00.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:00.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:01.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:01.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:02.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:02.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:03.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:03.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:05.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:05.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:06.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:06.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:06.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:06.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:08.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:08.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:09.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:09.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:10.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:10.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:11.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:11.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:12.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:12.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:13.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:13.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:14.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:14.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:15.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:15.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:16.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:16.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:16.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:16.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:17.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:17.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:18.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:18.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:19.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:19.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:20.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:20.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:21.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:21.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:23.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:23.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:24.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:23.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:24.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:24.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:26.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:26.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:27.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:27.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:28.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:28.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:29.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:29.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:30.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:30.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:31.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:31.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:32.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:32.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:33.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:33.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:33.153 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:19:33.153 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:19:33.153 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:19:33.154 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:19:33.154 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:19:33.160 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:19:33.160 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T03:19:33.161 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T03:19:33.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:19:34.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:34.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:35.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:35.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:36.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:36.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:37.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:37.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:38.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:38.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:39.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:39.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:40.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:40.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:41.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:41.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:42.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:42.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:43.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:43.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:44.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:44.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:44.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:44.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:45.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:45.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:46.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:46.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:47.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:47.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:48.162 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T03:19:48.162 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T03:19:48.163 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:19:48.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:48.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:19:48.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:48.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:49.189 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:19:49.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:19:49.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:19:49.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:49.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:19:49.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:49.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:50.232 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":256,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:50.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:50.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:19:50.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:50.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:51.261 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":256,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:51.285 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:51.285 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:19:51.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:51.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:52.305 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":256,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:52.323 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:19:52.324 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:52.324 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:19:53.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:52.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:53.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:19:54.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:54.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:54.356 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:19:54.356 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:54.356 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:19:55.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:55.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:55.308 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:19:55.338 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:19:55.338 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:19:55.338 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:19:55.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:55.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:19:56.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:56.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:56.333 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":256,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:56.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:56.354 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:19:57.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:57.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:57.338 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":256,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:57.360 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:57.360 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:19:58.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:58.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:58.324 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":256,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:19:58.346 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:19:58.346 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:19:58.347 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:19:58.347 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:19:58.347 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:19:58.347 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:19:58.365 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:19:58.365 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:19:58.366 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:19:58.366 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:19:59.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:19:59.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:19:59.404 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:19:59.435 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:00.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:00.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:01.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:01.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:02.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:02.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:02.436 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:20:02.436 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:20:03.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:03.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:03.448 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:03.470 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:04.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:04.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:05.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:05.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:06.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:06.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:06.471 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:20:06.471 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:20:07.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:07.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:07.453 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:07.476 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:08.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:08.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:09.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:09.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:10.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:10.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:10.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:20:10.477 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:20:11.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:11.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:12.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:12.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:13.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:13.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:14.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:14.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:14.520 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:14.543 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:20:14.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:20:14.543 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:20:14.543 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:20:15.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:15.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:15.551 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":260,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:20:15.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:20:15.582 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:20:16.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:16.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:16.561 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":260,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:20:16.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:20:17.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:17.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:18.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:18.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:19.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:19.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:20.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:20.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:21.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:21.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:22.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:22.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:23.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:23.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:24.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:24.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:25.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:25.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:26.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:26.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:27.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:27.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:28.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:28.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:29.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:29.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:30.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:30.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:31.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:31.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:32.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:32.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:33.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:33.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:34.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:34.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:35.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:35.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:36.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:36.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:37.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:37.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:38.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:38.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:39.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:39.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:40.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:40.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:41.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:41.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:42.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:42.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:43.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:43.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:44.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:44.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:45.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:45.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:46.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:46.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:47.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:47.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:48.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:48.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:49.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:49.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:50.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:50.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:51.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:51.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:52.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:52.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:53.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:53.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:54.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:54.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:55.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:55.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:56.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:56.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:57.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:57.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:58.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:58.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:20:59.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:20:59.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:00.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:00.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:01.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:01.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:02.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:02.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:03.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:03.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:04.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:04.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:05.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:05.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:06.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:06.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:07.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:07.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:08.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:08.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:09.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:09.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:10.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:10.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:11.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:11.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:12.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:12.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:13.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:13.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:14.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:14.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:15.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:15.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:16.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:16.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:17.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:17.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:18.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:18.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:19.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:19.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:20.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:20.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:21.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:21.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:22.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:22.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:23.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:23.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:23.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:23.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:24.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:24.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:25.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:25.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:26.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:26.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:27.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:27.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:28.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:28.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:29.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:29.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:30.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:30.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:31.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:31.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:32.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:32.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:33.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:33.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:34.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:34.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:35.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:35.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:36.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:36.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:37.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:37.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:38.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:38.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:39.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:39.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:40.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:40.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:41.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:41.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:42.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:42.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:43.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:43.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:44.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:44.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:45.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:45.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:46.577 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:21:46.578 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:21:46.578 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:21:46.578 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:21:46.578 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:21:46.587 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:21:46.588 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:21:46.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:21:46.588 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:21:46.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:46.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:47.585 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:21:47.607 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:21:47.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:47.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:48.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:48.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:49.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:49.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:50.608 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:21:50.608 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:21:50.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:50.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:51.610 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:21:51.634 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":260,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:21:51.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:51.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:52.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:52.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:53.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:53.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:54.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:21:54.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:21:54.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:54.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:55.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:55.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:56.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:56.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:57.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:57.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:58.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:58.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:21:59.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:21:59.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:00.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:00.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:01.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:01.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:01.854 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:01.876 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:01.876 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:22:01.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:01.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:22:02.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:02.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:02.818 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":268,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:02.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:02.841 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:22:03.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:03.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:03.858 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":268,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:03.888 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:03.888 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:22:04.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:04.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:04.895 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":268,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:04.925 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:22:04.925 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:04.925 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:22:05.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:05.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:06.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:22:06.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:06.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:07.058 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:22:07.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:07.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:22:07.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:07.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:08.053 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:08.084 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:08.084 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:22:08.084 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:22:08.084 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:08.084 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:22:08.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:08.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:09.097 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":268,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:09.120 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:09.120 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:22:09.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:09.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:10.135 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":268,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:10.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:10.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:22:10.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:10.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:11.139 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":268,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:11.160 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:22:11.160 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:22:11.160 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:22:11.161 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:22:11.161 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:22:11.161 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:22:11.187 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:22:11.187 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:22:11.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:11.187 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:22:11.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:11.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:12.250 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:12.280 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:12.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:12.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:13.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:13.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:14.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:14.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:15.282 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:15.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:22:15.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:15.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:16.287 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:16.314 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:16.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:16.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:17.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:17.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:18.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:18.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:19.316 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:19.316 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:22:19.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:19.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:20.323 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:20.354 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":268,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:20.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:20.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:21.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:21.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:22.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:22.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:23.355 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:23.355 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:22:23.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:23.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:24.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:24.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:25.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:25.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:26.621 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:26.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:26.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:26.652 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:22:26.652 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:22:26.653 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:26.653 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:22:27.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:27.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:27.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":272,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:27.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:22:27.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:22:28.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:28.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:28.677 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":272,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:22:28.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:22:29.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:29.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:30.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:30.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:31.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:31.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:32.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:32.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:33.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:33.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:34.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:34.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:35.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:35.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:36.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:36.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:37.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:37.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:38.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:38.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:39.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:39.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:40.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:40.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:41.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:41.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:42.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:42.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:43.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:43.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:44.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:44.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:45.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:45.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:46.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:46.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:47.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:47.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:48.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:48.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:49.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:49.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:50.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:50.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:51.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:51.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:52.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:52.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:53.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:53.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:54.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:54.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:55.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:55.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:56.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:56.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:57.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:57.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:58.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:58.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:22:59.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:22:59.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:00.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:00.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:01.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:01.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:02.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:02.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:03.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:03.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:04.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:04.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:05.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:05.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:06.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:06.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:07.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:07.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:08.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:08.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:09.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:09.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:10.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:10.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:11.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:11.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:12.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:12.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:13.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:13.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:14.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:14.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:15.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:15.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:16.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:16.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:17.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:17.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:18.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:18.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:19.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:19.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:20.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:20.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:21.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:21.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:22.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:22.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:23.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:23.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:24.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:24.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:25.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:25.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:26.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:26.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:27.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:27.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:28.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:28.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:29.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:29.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:30.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:30.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:31.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:31.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:32.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:32.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:33.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:33.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:34.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:34.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:35.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:35.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:36.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:36.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:37.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:37.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:38.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:38.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:39.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:39.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:40.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:40.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:41.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:41.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:42.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:42.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:43.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:43.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:44.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:44.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:45.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:45.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:46.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:46.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:47.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:47.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:48.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:48.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:49.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:49.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:50.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:50.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:51.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:51.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:52.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:52.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:53.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:53.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:54.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:54.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:55.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:55.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:56.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:56.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:57.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:57.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:58.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:58.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:58.705 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:23:58.705 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:23:58.705 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:23:58.705 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:23:58.705 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:23:58.711 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:23:58.711 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:23:58.711 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:23:58.712 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:23:59.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:23:59.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:23:59.699 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:23:59.722 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:00.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:00.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:01.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:01.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:02.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:02.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:02.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:02.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:03.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:03.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:03.718 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:03.748 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":272,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:04.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:04.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:05.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:05.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:06.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:06.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:06.749 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:06.749 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:07.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:07.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:07.707 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:07.729 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:07.731 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:24:07.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:07.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:24:08.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:08.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:08.701 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":278,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:08.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:08.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:24:09.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:09.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:09.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":278,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:09.716 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:09.716 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:24:10.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:10.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:10.690 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":278,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:10.720 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:24:10.720 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:10.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:24:11.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:11.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:11.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:24:12.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:12.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:12.859 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:24:12.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:12.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:13.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:13.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:13.860 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:13.892 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:13.892 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:24:13.893 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:24:13.893 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:13.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:24:14.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:14.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:14.934 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":278,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:14.954 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:14.955 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:24:15.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:15.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:15.937 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":278,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:15.959 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:15.959 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:24:16.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:16.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:16.969 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":278,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:16.991 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:24:16.992 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:24:16.992 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:24:16.992 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:24:16.993 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:24:16.993 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:24:17.013 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:24:17.013 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:24:17.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:17.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:17.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:17.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:18.088 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:18.119 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:18.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:18.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:19.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:19.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:20.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:20.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:21.120 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:21.120 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:21.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:21.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:22.097 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:22.120 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":278,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:22.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:22.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:23.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:23.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:24.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:24.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:25.121 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:25.121 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:24:25.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:25.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:26.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:26.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:27.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:27.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:28.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:28.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:29.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:29.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:30.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:30.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:31.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:31.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:32.372 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:32.395 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:24:32.395 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:24:32.395 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:32.395 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:24:32.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:32.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:33.413 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":282,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:33.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:24:33.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:24:33.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:33.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:34.470 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":282,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:24:34.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:24:34.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:34.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:35.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:35.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:36.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:36.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:37.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:37.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:38.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:38.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:39.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:39.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:40.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:40.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:41.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:41.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:42.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:42.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:43.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:43.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:44.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:44.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:45.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:45.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:46.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:46.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:47.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:47.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:48.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:48.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:49.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:49.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:50.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:50.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:51.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:51.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:52.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:52.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:53.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:53.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:54.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:54.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:55.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:55.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:56.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:56.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:57.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:57.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:58.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:58.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:24:59.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:24:59.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:00.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:00.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:01.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:01.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:02.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:02.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:03.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:03.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:04.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:04.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:05.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:05.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:06.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:06.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:07.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:07.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:08.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:08.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:09.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:09.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:10.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:10.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:11.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:11.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:12.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:12.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:13.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:13.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:14.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:14.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:15.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:15.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:16.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:16.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:17.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:17.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:18.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:18.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:19.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:19.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:20.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:20.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:21.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:21.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:22.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:22.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:23.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:23.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:24.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:24.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:25.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:25.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:26.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:26.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:27.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:27.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:28.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:28.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:29.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:29.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:30.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:30.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:31.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:31.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:32.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:32.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:33.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:33.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:34.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:34.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:35.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:35.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:36.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:36.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:37.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:37.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:38.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:38.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:39.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:39.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:40.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:40.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:41.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:41.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:42.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:42.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:43.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:43.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:44.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:44.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:45.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:45.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:46.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:46.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:47.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:47.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:48.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:48.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:49.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:49.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:50.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:50.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:51.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:51.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:52.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:52.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:53.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:53.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:54.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:54.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:55.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:55.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:56.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:56.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:57.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:57.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:58.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:58.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:25:59.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:25:59.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:00.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:00.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:01.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:01.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:02.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:02.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:03.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:03.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:04.501 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:26:04.501 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:26:04.501 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:26:04.502 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:04.502 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:26:04.507 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:26:04.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:26:04.507 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:04.507 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:04.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:04.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:05.452 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:05.473 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:05.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:05.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:06.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:06.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:07.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:07.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:08.475 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:08.475 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:08.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:08.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:09.450 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:09.479 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:09.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:09.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:10.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:10.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:11.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:11.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:12.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:12.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:12.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:12.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:13.518 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:13.549 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":282,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:13.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:13.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:14.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:14.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:15.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:15.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:16.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:16.551 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:16.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:16.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:17.563 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:17.595 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:17.596 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:26:17.596 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:17.596 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:26:17.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:17.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:18.568 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":284,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:18.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:18.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:26:18.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:18.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:19.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:19.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:19.667 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":284,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:19.692 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:19.692 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:26:20.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:20.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:20.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":284,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:20.709 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:26:20.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:20.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:26:21.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:21.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:21.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:26:22.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:22.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:22.804 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:26:22.804 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:22.804 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:23.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:23.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:24.046 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:24.068 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:24.069 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:26:24.069 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:26:24.069 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:24.069 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:26:24.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:24.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:25.027 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":284,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:25.049 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:25.049 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:26:25.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:25.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:26.067 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":284,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:26.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:26.099 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:26:26.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:26.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:27.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":284,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:27.133 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:26:27.133 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:26:27.133 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:26:27.133 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:26:27.134 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:26:27.134 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:26:27.152 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:26:27.152 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:26:27.152 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:27.152 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:27.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:27.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:28.197 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:28.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:28.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:28.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:29.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:29.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:30.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:30.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:31.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:31.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:31.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:31.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:32.223 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:32.255 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:32.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:32.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:33.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:33.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:34.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:34.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:35.256 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:35.256 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:35.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:35.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:36.240 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:36.263 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":284,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:36.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:36.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:37.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:37.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:38.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:38.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:39.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:39.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:26:39.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:39.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:40.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:40.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:41.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:41.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:42.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:42.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:42.704 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":288,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:42.728 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":288,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:26:42.728 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:26:42.728 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:42.728 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:26:43.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:43.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:43.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":288,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:26:43.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:26:43.747 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:26:44.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:44.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:44.767 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":288,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:26:44.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:26:45.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:45.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:46.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:46.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:47.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:47.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:48.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:48.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:49.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:49.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:50.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:50.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:51.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:51.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:52.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:52.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:53.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:53.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:54.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:54.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:55.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:55.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:56.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:56.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:57.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:57.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:58.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:58.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:26:59.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:26:59.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:00.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:00.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:01.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:01.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:02.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:02.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:03.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:03.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:04.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:04.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:05.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:05.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:06.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:06.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:07.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:07.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:08.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:08.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:09.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:09.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:10.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:10.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:11.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:11.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:12.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:12.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:13.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:13.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:14.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:14.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:15.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:15.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:16.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:16.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:17.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:17.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:18.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:18.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:19.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:19.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:20.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:20.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:21.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:21.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:22.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:22.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:23.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:23.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:24.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:24.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:25.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:25.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:26.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:26.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:27.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:27.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:28.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:28.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:29.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:29.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:30.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:30.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:31.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:31.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:32.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:32.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:33.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:33.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:34.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:34.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:35.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:35.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:36.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:36.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:37.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:37.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:38.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:38.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:39.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:39.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:40.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:40.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:41.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:41.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:42.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:42.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:43.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:43.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:44.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:44.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:45.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:45.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:46.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:46.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:47.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:47.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:48.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:48.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:49.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:49.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:50.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:50.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:51.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:51.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:52.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:52.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:53.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:53.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:54.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:54.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:55.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:55.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:56.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:56.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:57.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:57.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:58.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:58.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:27:59.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:27:59.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:00.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:00.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:01.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:01.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:02.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:02.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:03.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:03.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:04.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:04.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:05.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:05.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:06.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:06.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:07.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:07.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:08.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:08.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:09.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:09.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:10.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:10.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:11.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:11.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:12.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:12.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:13.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:13.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:14.798 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:28:14.798 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:28:14.798 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:28:14.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:14.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:28:14.804 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:28:14.805 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:28:14.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:14.805 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:14.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:14.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:15.789 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":288,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:15.812 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":288,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:15.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:15.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:16.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:16.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:17.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:17.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:18.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:18.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:18.813 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:18.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:19.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:19.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:20.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:20.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:21.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:21.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:22.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:22.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:23.743 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:23.770 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:23.770 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:28:23.770 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:23.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:28:23.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:23.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:24.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:24.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:24.793 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":294,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:24.816 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:24.816 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:28:25.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:25.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:25.864 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":294,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:25.893 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:25.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:28:26.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:26.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:26.897 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":294,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:26.920 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:28:26.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:26.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:28:27.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:27.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:27.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:28:28.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:28.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:28.935 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:28:28.936 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:28.936 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:29.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:29.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:29.870 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:29.900 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:29.900 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:28:29.900 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:28:29.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:29.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:28:30.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:30.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:30.891 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":294,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:30.923 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:30.923 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:28:31.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:31.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:31.910 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":294,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:31.944 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:31.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:28:32.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:32.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:32.903 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":294,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:32.925 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:28:32.925 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:28:32.925 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:28:32.925 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:28:32.925 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:28:32.926 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:28:32.942 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:28:32.942 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:28:32.942 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:32.942 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:33.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:33.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:33.995 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:34.017 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:34.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:34.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:35.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:35.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:36.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:36.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:37.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:37.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:37.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:37.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:38.024 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:38.047 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:38.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:38.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:39.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:39.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:40.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:40.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:41.049 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:41.049 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:28:41.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:41.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:42.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:42.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:43.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:43.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:44.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:44.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:45.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:45.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:46.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:46.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:46.986 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":298,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:47.018 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":298,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:28:47.018 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:28:47.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:47.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:28:47.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:47.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:48.045 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":298,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:48.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:28:48.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:28:48.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:48.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:49.093 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":298,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:28:49.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:28:49.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:49.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:50.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:50.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:51.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:51.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:52.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:52.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:53.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:53.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:54.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:54.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:55.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:55.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:56.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:56.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:57.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:57.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:58.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:58.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:28:59.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:28:59.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:00.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:00.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:01.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:01.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:02.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:02.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:03.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:03.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:04.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:04.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:05.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:05.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:06.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:06.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:07.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:07.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:08.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:08.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:09.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:09.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:10.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:10.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:11.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:11.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:12.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:12.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:13.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:13.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:14.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:14.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:15.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:15.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:16.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:16.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:17.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:17.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:18.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:18.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:19.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:19.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:20.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:20.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:21.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:21.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:22.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:22.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:23.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:23.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:24.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:24.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:25.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:25.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:26.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:26.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:27.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:27.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:28.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:28.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:29.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:29.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:30.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:30.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:31.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:31.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:32.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:32.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:33.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:33.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:34.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:34.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:35.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:35.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:36.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:36.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:37.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:37.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:38.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:38.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:39.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:39.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:40.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:40.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:41.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:41.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:42.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:42.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:43.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:43.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:44.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:44.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:45.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:45.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:46.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:46.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:47.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:47.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:48.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:48.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:49.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:49.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:50.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:50.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:51.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:51.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:52.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:52.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:53.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:53.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:54.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:54.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:55.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:55.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:56.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:56.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:57.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:57.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:58.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:58.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:29:59.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:29:59.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:00.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:00.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:01.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:01.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:02.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:02.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:03.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:03.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:04.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:04.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:05.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:05.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:06.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:06.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:07.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:07.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:08.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:08.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:09.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:09.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:10.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:10.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:11.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:11.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:12.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:12.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:13.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:13.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:14.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:14.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:15.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:15.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:16.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:16.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:17.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:17.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:18.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:18.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:19.115 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:30:19.115 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:30:19.116 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:30:19.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:19.116 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:30:19.121 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:30:19.121 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:30:19.122 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:19.122 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:19.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:19.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:20.089 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":298,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:20.122 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":298,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:20.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:20.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:21.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:21.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:22.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:22.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:23.123 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:23.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:23.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:23.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:24.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:24.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:25.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:25.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:26.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:26.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:27.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:27.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:28.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:28.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:28.629 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:28.653 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:28.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:30:28.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:28.654 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:30:29.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:29.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:29.723 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":304,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:29.746 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:29.746 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:30:30.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:30.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:30.734 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":304,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:30.757 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:30.757 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:30:31.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:31.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:31.763 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":304,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:31.785 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:30:31.785 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:31.785 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:30:32.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:32.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:32.874 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:30:33.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:33.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:33.875 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:30:33.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:33.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:34.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:34.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:34.841 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:34.871 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:34.871 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:30:34.872 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:30:34.872 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:34.872 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:30:35.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:35.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:35.877 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":304,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:35.901 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:35.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:30:36.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:36.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:36.882 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":304,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:36.908 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:36.909 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:30:37.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:37.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:37.857 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":304,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:37.880 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:30:37.880 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:30:37.880 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:30:37.880 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:30:37.880 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:30:37.881 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:30:37.897 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:30:37.897 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:30:37.897 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:37.897 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:38.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:38.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:38.943 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:38.974 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:39.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:39.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:40.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:40.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:41.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:41.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:41.975 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:41.975 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:42.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:42.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:42.965 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:42.995 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:43.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:43.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:44.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:44.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:45.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:45.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:45.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:45.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:30:46.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:46.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:47.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:47.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:48.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:48.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:49.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:49.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:50.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:50.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:51.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:51.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:52.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:52.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:53.405 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":308,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:53.436 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":308,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:30:53.437 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:30:53.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:53.437 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:30:53.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:53.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:54.430 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":308,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:54.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:30:54.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:30:54.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:54.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:55.502 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":308,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:30:55.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:55.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:55.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:30:56.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:56.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:57.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:57.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:58.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:58.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:30:59.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:30:59.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:00.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:00.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:01.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:01.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:02.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:02.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:03.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:03.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:04.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:04.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:05.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:05.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:06.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:06.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:07.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:07.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:08.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:08.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:09.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:09.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:10.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:10.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:11.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:11.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:12.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:12.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:13.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:13.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:14.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:14.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:15.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:15.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:16.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:16.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:17.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:17.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:18.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:18.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:19.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:19.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:20.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:20.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:21.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:21.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:22.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:22.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:23.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:23.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:24.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:24.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:25.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:25.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:26.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:26.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:27.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:27.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:28.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:28.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:29.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:29.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:30.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:30.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:31.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:31.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:32.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:32.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:33.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:33.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:34.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:34.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:35.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:35.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:36.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:36.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:37.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:37.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:38.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:38.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:39.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:39.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:40.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:40.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:41.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:41.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:42.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:42.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:43.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:43.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:44.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:44.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:45.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:45.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:46.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:46.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:47.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:47.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:48.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:48.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:49.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:49.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:50.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:50.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:51.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:51.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:52.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:52.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:53.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:53.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:54.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:54.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:55.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:55.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:56.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:56.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:57.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:57.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:58.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:58.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:31:59.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:31:59.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:00.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:00.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:01.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:01.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:02.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:02.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:03.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:03.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:04.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:04.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:05.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:05.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:06.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:06.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:07.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:07.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:08.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:08.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:09.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:09.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:10.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:10.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:11.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:11.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:12.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:12.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:13.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:13.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:14.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:14.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:15.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:15.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:16.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:16.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:17.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:17.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:18.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:18.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:19.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:19.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:20.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:20.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:21.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:21.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:22.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:22.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:23.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:23.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:24.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:24.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:25.532 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:32:25.533 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:32:25.533 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:32:25.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:25.533 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:32:25.540 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:32:25.540 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:32:25.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:25.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:32:25.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:25.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:26.491 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":308,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:26.513 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":308,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:26.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:26.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:27.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:27.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:28.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:28.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:29.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:29.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:32:29.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:29.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:30.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:30.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:31.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:31.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:32.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:32.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:33.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:33.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:34.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:34.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:35.046 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":312,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:35.069 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":312,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:35.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:35.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:36.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:36.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:37.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:37.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:38.070 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:38.070 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:32:38.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:38.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:39.078 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":314,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:39.100 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":314,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:39.101 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:32:39.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:39.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:32:39.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:39.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:40.066 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":314,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:40.096 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:40.096 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:32:40.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:40.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:41.073 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":314,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:41.098 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:41.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:32:41.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:41.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:42.103 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":314,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:42.125 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:32:42.126 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:42.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:32:42.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:42.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:43.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:32:43.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:43.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:44.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:32:44.202 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:44.202 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:32:44.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:44.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:45.194 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":314,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:45.216 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":314,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:32:45.217 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:32:45.217 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:32:45.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:45.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:32:45.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:45.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:46.225 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":314,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:46.247 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:46.247 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:32:46.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:46.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:47.264 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":314,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:47.295 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:32:47.295 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:32:47.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:47.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:48.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:48.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:48.351 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":314,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:32:48.372 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:32:48.372 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T03:32:48.372 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:32:48.372 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:32:48.373 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:32:48.373 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:32:48.411 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:32:48.412 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T03:32:48.412 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T03:32:48.412 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:32:49.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:49.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:50.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:50.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:51.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:51.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:52.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:52.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:53.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:53.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:54.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:54.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:55.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:55.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:56.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:56.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:57.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:57.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:58.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:58.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:32:59.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:32:59.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:00.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:00.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:01.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:01.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:02.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:02.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:03.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:03.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:03.413 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T03:33:03.413 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T03:33:03.413 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:33:03.414 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:33:03.414 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:33:04.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:04.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:05.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:05.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:06.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:06.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:07.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:07.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:08.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:08.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:09.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:09.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:10.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:10.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:10.672 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":316,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:33:10.696 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":316,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:33:10.696 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:33:10.696 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:33:10.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:33:11.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:11.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:11.667 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":316,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:33:11.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:33:11.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:33:12.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:12.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:12.718 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":316,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:33:12.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:33:13.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:13.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:14.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:14.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:15.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:15.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:16.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:16.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:17.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:17.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:18.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:18.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:19.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:19.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:20.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:20.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:21.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:21.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:22.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:22.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:23.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:23.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:24.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:24.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:25.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:25.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:26.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:26.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:27.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:27.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:28.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:28.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:29.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:29.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:30.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:30.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:30.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:30.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:31.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:31.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:32.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:32.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:33.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:33.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:34.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:34.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:35.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:35.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:36.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:36.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:37.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:37.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:38.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:38.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:39.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:39.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:40.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:40.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:41.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:41.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:42.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:42.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:43.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:43.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:44.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:44.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:45.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:45.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:46.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:46.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:47.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:47.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:48.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:48.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:49.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:49.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:50.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:50.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:51.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:51.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:52.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:52.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:53.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:53.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:54.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:54.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:55.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:55.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:56.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:56.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:57.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:57.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:58.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:58.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:33:59.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:33:59.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:00.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:00.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:01.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:01.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:02.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:02.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:03.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:03.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:04.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:04.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:05.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:05.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:06.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:06.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:07.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:07.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:08.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:08.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:09.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:09.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:10.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:10.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:11.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:11.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:12.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:12.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:13.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:13.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:14.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:14.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:15.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:15.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:16.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:16.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:17.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:17.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:18.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:18.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:19.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:19.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:20.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:20.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:21.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:21.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:22.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:22.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:23.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:23.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:24.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:24.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:25.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:25.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:26.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:26.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:27.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:27.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:28.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:28.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:29.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:29.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:30.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:30.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:31.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:31.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:32.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:32.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:33.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:33.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:34.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:34.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:35.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:35.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:36.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:36.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:37.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:37.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:38.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:38.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:39.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:39.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:40.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:40.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:41.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:41.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:42.747 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:34:42.747 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:34:42.747 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:34:42.748 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:34:42.748 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:34:42.756 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:34:42.756 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T03:34:42.757 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T03:34:42.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:34:42.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:42.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:43.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:43.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:44.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:44.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:45.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:45.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:46.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:46.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:47.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:47.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:48.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:48.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:49.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:49.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:50.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:50.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:51.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:51.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:52.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:52.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:53.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:53.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:54.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:54.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:55.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:55.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:56.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:56.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:57.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:57.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:57.757 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T03:34:57.758 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T03:34:57.758 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:34:57.758 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:34:57.758 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:34:58.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:58.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:34:58.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":324,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:34:59.018 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":324,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:34:59.019 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:34:59.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:34:59.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:34:59.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:34:59.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:00.014 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":324,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:00.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:00.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:35:00.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:00.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:01.030 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":324,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:01.060 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:01.061 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:35:01.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:01.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:02.087 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":324,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:02.109 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:35:02.110 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:02.110 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:35:02.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:02.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:03.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:35:03.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:03.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:04.234 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:35:04.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:04.235 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:35:04.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:04.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:05.227 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":324,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:35:05.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":324,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:35:05.259 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:35:05.259 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:35:05.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:05.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:35:05.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:05.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:06.261 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":324,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:06.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:06.291 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:35:06.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:06.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:07.234 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":324,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:07.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:07.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:35:07.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:07.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:08.201 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":324,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:08.224 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:35:08.224 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'b'] 2019-10-01T03:35:08.224 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:35:08.224 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:35:08.225 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:35:08.225 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:35:08.253 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:35:08.253 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T03:35:08.254 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T03:35:08.254 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:35:08.254 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:35:08.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:35:08.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:08.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:09.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:09.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:10.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:10.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:11.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:11.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:12.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:12.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:13.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:13.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:14.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:14.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:15.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:15.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:16.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:16.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:17.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:17.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:18.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:18.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:19.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:19.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:20.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:20.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:21.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:21.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:22.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:22.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:23.255 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T03:35:23.255 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T03:35:23.256 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:35:23.256 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:35:23.256 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:35:23.256 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:23.256 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:35:23.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:23.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:24.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:24.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:25.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:25.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:26.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:26.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:27.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:27.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:28.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:28.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:29.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:29.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:30.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:30.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:30.486 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":328,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:35:30.508 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":328,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:35:30.508 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:35:30.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:30.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:35:31.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:31.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:31.474 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":328,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:31.504 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:35:31.505 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:35:32.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:32.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:32.509 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":328,"quorum":[0,1],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:35:32.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:35:33.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:33.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:34.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:34.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:35.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:35.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:36.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:36.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:37.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:37.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:38.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:38.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:39.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:39.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:40.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:40.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:41.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:41.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:42.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:42.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:43.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:43.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:44.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:44.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:45.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:45.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:46.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:46.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:47.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:47.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:48.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:48.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:49.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:49.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:50.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:50.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:51.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:51.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:52.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:52.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:53.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:53.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:54.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:54.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:55.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:55.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:56.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:56.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:57.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:57.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:58.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:58.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:35:59.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:35:59.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:00.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:00.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:01.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:01.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:02.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:02.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:03.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:03.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:04.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:04.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:05.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:05.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:06.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:06.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:07.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:07.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:08.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:08.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:09.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:09.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:10.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:10.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:11.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:11.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:12.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:12.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:13.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:13.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:14.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:14.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:15.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:15.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:16.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:16.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:17.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:17.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:18.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:18.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:19.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:19.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:20.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:20.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:21.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:21.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:22.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:22.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:23.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:23.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:24.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:24.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:25.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:25.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:26.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:26.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:27.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:27.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:28.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:28.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:29.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:29.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:30.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:30.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:31.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:31.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:32.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:32.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:33.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:33.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:34.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:34.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:35.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:35.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:36.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:36.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:37.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:37.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:38.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:38.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:39.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:39.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:40.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:40.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:41.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:41.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:42.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:42.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:43.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:43.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:44.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:44.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:45.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:45.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:46.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:46.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:47.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:47.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:48.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:48.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:49.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:49.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:50.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:50.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:51.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:51.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:52.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:52.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:53.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:53.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:54.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:54.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:55.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:55.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:56.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:56.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:57.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:57.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:58.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:58.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:36:59.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:36:59.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:00.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:00.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:01.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:01.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:02.531 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:37:02.532 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:37:02.532 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:37:02.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:02.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:37:02.538 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:37:02.539 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T03:37:02.539 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T03:37:02.539 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T03:37:02.539 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T03:37:02.540 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T03:37:02.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:02.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:03.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:03.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:04.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:04.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:05.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:05.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:06.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:06.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:07.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:07.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:08.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:08.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:09.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:09.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:10.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:10.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:11.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:11.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:12.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:12.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:13.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:13.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:14.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:14.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:15.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:15.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:16.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:16.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:17.546 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T03:37:17.547 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T03:37:17.547 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T03:37:17.548 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T03:37:17.548 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:37:17.548 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:17.548 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:17.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:17.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:18.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:18.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:19.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:19.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:20.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:20.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:21.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:21.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:22.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:22.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:23.195 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:23.226 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:23.226 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:37:23.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:23.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:37:23.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:23.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:24.291 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":334,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:24.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:24.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:37:24.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:24.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:25.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:25.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:26.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:26.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:27.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:27.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:28.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:28.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:29.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:29.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:30.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:30.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:31.514 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":334,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:31.545 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:31.545 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:37:31.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:31.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:32.538 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":334,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:32.561 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:37:32.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:32.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:37:32.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:32.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:33.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:37:33.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:33.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:34.623 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:37:34.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:34.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:34.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:34.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:35.639 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:35.670 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:35.670 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:37:35.670 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:37:35.671 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:35.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:37:35.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:35.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:36.673 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":334,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:36.703 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:36.703 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:37:36.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:36.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:37.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:37.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:37.803 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":334,"quorum":[0,1,2],"quorum_age":14,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:37.826 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:37.826 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:37:38.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:38.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:38.840 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":334,"quorum":[0,1,2],"quorum_age":15,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:38.862 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:37:38.862 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:37:38.862 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:37:38.863 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:37:38.863 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:37:38.863 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:37:38.903 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:37:38.903 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:37:38.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:38.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:39.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:39.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:39.955 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:39.987 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:40.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:40.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:41.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:41.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:42.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:42.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:42.988 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:42.989 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:43.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:43.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:43.973 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:43.995 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:44.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:44.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:45.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:45.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:46.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:46.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:46.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:46.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:47.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:47.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:48.004 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":24,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:48.034 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":334,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":24,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:48.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:48.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:49.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:49.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:50.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:50.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:51.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:51.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:37:51.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:51.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:52.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:52.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:53.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:53.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:54.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:54.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:55.090 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":338,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:55.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":338,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:37:55.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:37:55.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:55.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:37:55.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:55.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:56.098 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":338,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:56.121 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:37:56.121 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:37:56.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:56.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:57.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":338,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:37:57.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:37:57.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:57.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:58.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:58.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:37:59.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:37:59.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:00.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:00.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:01.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:01.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:02.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:02.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:03.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:03.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:04.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:04.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:05.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:05.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:06.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:06.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:07.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:07.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:08.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:08.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:09.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:09.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:10.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:10.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:11.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:11.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:12.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:12.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:13.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:13.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:14.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:14.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:15.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:15.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:16.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:16.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:17.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:17.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:18.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:18.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:19.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:19.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:20.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:20.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:21.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:21.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:22.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:22.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:23.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:23.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:24.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:24.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:25.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:25.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:26.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:26.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:27.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:27.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:28.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:28.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:29.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:29.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:30.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:30.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:31.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:31.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:32.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:32.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:33.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:33.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:34.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:34.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:35.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:35.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:36.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:36.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:37.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:37.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:38.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:38.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:39.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:39.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:40.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:40.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:41.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:41.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:42.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:42.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:43.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:43.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:44.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:44.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:45.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:45.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:46.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:46.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:47.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:47.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:48.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:48.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:49.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:49.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:50.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:50.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:51.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:51.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:52.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:52.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:53.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:53.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:54.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:54.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:55.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:55.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:56.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:56.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:57.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:57.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:58.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:58.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:38:59.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:38:59.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:00.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:00.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:01.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:01.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:02.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:02.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:03.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:03.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:04.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:04.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:05.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:05.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:06.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:06.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:07.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:07.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:08.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:08.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:09.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:09.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:10.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:10.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:11.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:11.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:12.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:12.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:13.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:13.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:14.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:14.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:15.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:15.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:16.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:16.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:17.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:17.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:18.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:18.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:19.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:19.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:20.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:20.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:21.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:21.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:22.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:22.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:23.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:23.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:24.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:24.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:25.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:25.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:26.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:26.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:27.146 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:39:27.146 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:39:27.146 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:39:27.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:27.147 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:39:27.152 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:39:27.153 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:39:27.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:27.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:27.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:27.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:28.123 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":338,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:28.146 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":338,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:28.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:28.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:29.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:29.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:30.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:30.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:31.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:31.148 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:31.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:31.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:32.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:32.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:33.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:33.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:34.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:34.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:35.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:35.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:36.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:36.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:37.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:37.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:38.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:38.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:39.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:39.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:40.119 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:40.136 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:40.137 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:39:40.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:40.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:39:40.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:40.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:41.094 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":350,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:41.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:41.116 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:39:41.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:41.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:42.123 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":350,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:42.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:42.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:39:42.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:42.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:43.194 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":350,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:43.217 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:39:43.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:43.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:39:43.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:43.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:44.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:39:44.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:44.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:45.333 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:39:45.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:45.334 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:45.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:45.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:46.334 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:46.358 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:46.358 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:39:46.359 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:39:46.359 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:46.359 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:39:46.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:46.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:47.380 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":350,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:47.402 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:47.402 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:39:47.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:47.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:48.410 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":350,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:48.441 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:48.441 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:39:48.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:48.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:49.396 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":350,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:39:49.427 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:39:49.427 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:39:49.427 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:39:49.427 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:39:49.427 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:39:49.428 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:39:49.446 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:39:49.446 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:39:49.446 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:49.447 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:49.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:49.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:50.523 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:50.545 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:50.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:50.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:51.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:51.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:52.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:52.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:53.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:53.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:53.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:53.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:54.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:54.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:54.564 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:54.587 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:55.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:55.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:56.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:56.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:57.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:57.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:57.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:39:57.588 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:39:58.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:58.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:39:58.565 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:58.587 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":350,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:39:59.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:39:59.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:00.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:00.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:01.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:01.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:01.589 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:40:01.589 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:40:02.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:02.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:03.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:03.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:04.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:04.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:05.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:05.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:05.650 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":354,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:40:05.672 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":354,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:40:05.672 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:40:05.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:40:05.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:40:06.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:06.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:06.658 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":354,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:40:06.689 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:40:06.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:40:07.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:07.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:07.710 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":354,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:40:07.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:40:08.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:08.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:09.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:09.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:10.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:10.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:11.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:11.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:12.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:12.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:13.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:13.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:14.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:14.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:15.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:15.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:16.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:16.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:17.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:17.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:18.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:18.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:19.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:19.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:20.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:20.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:21.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:21.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:22.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:22.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:23.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:23.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:24.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:24.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:25.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:25.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:26.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:26.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:27.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:27.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:28.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:28.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:29.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:29.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:30.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:30.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:31.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:31.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:32.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:32.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:33.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:33.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:34.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:34.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:35.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:35.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:36.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:36.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:37.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:37.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:38.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:38.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:39.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:39.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:40.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:40.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:41.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:41.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:42.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:42.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:43.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:43.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:44.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:44.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:45.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:45.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:46.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:46.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:47.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:47.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:48.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:48.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:49.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:49.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:50.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:50.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:51.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:51.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:52.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:52.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:53.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:53.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:54.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:54.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:55.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:55.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:56.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:56.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:57.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:57.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:58.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:58.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:40:59.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:40:59.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:00.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:00.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:01.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:01.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:02.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:02.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:03.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:03.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:04.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:04.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:05.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:05.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:06.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:06.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:07.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:07.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:08.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:08.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:09.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:09.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:10.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:10.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:11.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:11.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:12.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:12.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:13.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:13.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:14.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:14.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:15.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:15.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:16.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:16.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:17.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:17.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:18.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:18.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:19.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:19.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:20.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:20.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:21.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:21.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:22.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:22.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:23.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:23.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:24.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:24.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:25.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:25.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:26.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:26.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:27.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:27.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:28.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:28.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:29.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:29.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:30.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:30.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:31.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:31.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:32.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:32.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:33.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:33.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:34.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:34.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:35.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:35.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:36.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:36.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:37.740 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:41:37.740 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:41:37.740 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:41:37.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:37.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:41:37.746 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:41:37.746 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:41:37.746 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:37.747 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:41:37.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:37.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:38.716 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":354,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:38.747 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":354,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:38.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:38.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:39.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:39.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:40.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:40.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:41.748 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:41.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:41:41.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:41.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:42.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:42.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:43.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:43.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:44.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:44.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:45.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:45.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:46.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:46.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:47.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:47.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:48.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:48.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:49.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:49.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:50.732 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:50.753 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:50.753 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:41:50.753 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:50.753 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:41:50.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:50.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:51.745 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":366,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:41:51.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:51.777 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:41:51.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:51.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:52.806 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":366,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:41:52.824 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:52.824 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:41:52.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:52.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:53.749 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":366,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:41:53.772 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:41:53.773 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:53.773 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:41:53.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:53.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:54.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:41:54.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:54.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:55.793 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:41:55.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:55.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:41:55.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:55.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:56.753 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:56.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:41:56.783 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:41:56.783 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:41:56.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:56.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:41:56.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:56.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:57.780 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":366,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:41:57.802 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:57.802 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:41:57.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:57.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:58.800 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":366,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:41:58.830 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:58.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:41:58.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:58.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:59.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:41:59.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:41:59.850 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":366,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:41:59.872 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:41:59.873 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:41:59.873 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:41:59.873 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:41:59.873 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:41:59.873 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:41:59.909 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:41:59.909 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:41:59.909 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:41:59.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:42:00.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:00.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:00.970 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:00.993 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:01.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:01.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:02.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:02.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:03.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:03.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:03.994 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:42:03.994 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:42:04.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:04.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:04.963 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:04.985 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:05.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:05.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:06.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:06.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:07.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:07.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:07.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:42:07.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:42:08.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:08.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:09.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:09.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:10.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:10.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:11.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:11.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:12.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:12.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:13.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:13.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:14.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:14.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:15.415 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:15.447 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:42:15.447 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:42:15.448 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:42:15.448 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:42:15.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:15.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:16.448 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":370,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:42:16.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:42:16.477 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:42:16.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:16.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:17.488 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":370,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:42:17.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:42:17.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:17.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:18.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:18.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:19.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:19.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:20.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:20.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:21.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:21.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:22.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:22.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:23.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:23.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:24.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:24.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:25.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:25.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:27.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:27.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:27.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:27.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:28.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:28.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:30.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:30.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:31.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:31.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:32.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:32.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:33.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:33.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:34.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:34.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:35.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:35.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:36.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:36.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:37.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:37.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:38.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:38.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:39.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:39.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:40.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:40.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:41.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:41.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:42.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:42.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:43.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:43.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:44.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:44.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:45.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:45.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:46.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:46.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:47.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:47.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:48.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:48.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:49.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:49.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:50.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:50.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:51.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:51.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:52.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:52.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:53.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:53.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:54.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:54.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:55.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:55.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:56.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:56.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:57.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:57.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:58.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:58.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:42:59.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:42:59.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:00.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:00.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:01.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:01.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:02.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:02.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:03.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:03.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:04.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:04.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:05.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:05.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:06.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:06.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:06.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:06.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:08.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:08.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:09.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:09.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:10.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:10.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:11.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:11.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:11.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:11.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:12.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:12.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:13.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:13.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:14.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:14.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:15.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:15.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:16.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:16.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:17.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:17.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:18.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:18.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:19.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:19.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:20.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:20.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:21.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:21.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:22.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:22.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:23.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:23.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:24.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:24.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:25.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:25.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:26.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:26.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:27.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:27.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:28.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:28.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:29.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:29.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:30.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:30.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:31.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:31.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:32.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:32.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:33.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:33.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:34.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:34.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:35.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:35.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:36.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:36.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:37.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:37.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:38.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:38.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:39.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:39.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:40.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:40.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:41.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:41.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:42.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:42.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:43.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:43.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:44.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:44.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:45.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:45.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:46.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:46.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:47.518 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:43:47.518 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:43:47.518 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:43:47.519 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:43:47.519 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:43:47.524 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:43:47.524 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:43:47.525 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:43:47.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:43:47.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:47.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:48.482 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:43:48.512 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:43:48.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:48.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:49.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:49.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:50.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:50.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:51.513 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:43:51.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:43:52.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:52.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:53.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:53.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:54.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:54.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:55.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:55.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:56.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:56.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:57.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:57.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:58.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:58.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:58.742 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:43:58.765 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:43:58.765 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:43:58.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:43:58.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:43:59.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:43:59.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:43:59.758 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":376,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:43:59.790 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:43:59.790 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:44:00.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:00.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:00.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":376,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:00.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:00.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:44:01.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:01.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:01.812 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":376,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:01.834 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:44:01.834 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:01.835 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:44:02.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:02.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:03.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:44:03.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:03.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:04.121 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:44:04.121 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:04.121 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:44:04.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:04.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:05.127 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:05.156 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:05.156 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:44:05.156 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:44:05.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:05.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:44:05.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:05.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:06.162 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":376,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:06.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:06.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:44:06.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:06.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:07.200 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":376,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:07.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:07.223 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:44:07.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:07.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:08.227 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":376,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:08.249 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:44:08.249 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:44:08.250 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:44:08.250 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:44:08.250 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:44:08.250 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:44:08.284 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:44:08.285 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:44:08.285 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:08.285 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:44:08.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:08.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:09.298 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:09.320 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:09.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:09.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:10.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:10.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:11.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:11.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:12.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:12.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:12.321 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:12.321 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:44:13.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:13.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:13.345 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:13.375 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:14.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:14.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:15.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:15.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:16.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:16.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:16.377 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:16.377 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:44:17.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:17.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:18.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:18.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:19.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:19.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:20.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:20.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:21.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:21.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:22.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:22.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:23.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:23.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:24.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:24.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:25.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:25.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:26.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:26.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:27.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:27.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:28.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:28.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:28.814 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:28.843 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:44:28.843 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:44:28.843 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:28.843 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:44:29.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:29.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:29.831 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":380,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:29.853 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:44:29.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:44:30.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:30.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:30.808 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":380,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:44:30.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:44:31.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:31.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:32.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:32.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:33.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:33.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:34.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:34.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:35.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:35.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:36.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:36.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:37.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:37.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:38.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:38.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:39.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:39.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:40.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:40.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:41.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:41.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:42.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:42.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:43.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:43.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:44.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:44.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:45.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:45.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:46.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:46.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:47.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:47.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:48.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:48.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:49.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:49.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:50.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:50.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:51.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:51.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:52.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:52.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:53.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:53.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:54.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:54.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:55.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:55.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:56.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:56.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:57.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:57.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:58.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:58.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:44:59.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:44:59.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:00.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:00.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:01.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:01.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:02.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:02.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:03.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:03.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:04.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:04.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:05.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:05.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:06.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:06.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:07.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:07.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:08.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:08.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:09.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:09.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:10.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:10.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:11.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:11.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:12.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:12.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:13.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:13.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:14.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:14.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:15.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:15.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:16.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:16.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:17.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:17.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:18.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:18.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:19.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:19.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:20.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:20.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:21.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:21.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:22.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:22.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:23.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:23.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:24.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:24.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:25.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:25.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:26.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:26.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:27.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:27.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:28.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:28.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:29.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:29.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:30.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:30.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:31.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:31.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:32.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:32.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:33.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:33.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:34.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:34.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:35.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:35.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:36.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:36.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:37.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:37.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:38.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:38.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:39.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:39.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:40.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:40.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:41.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:41.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:42.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:42.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:43.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:43.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:44.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:44.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:45.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:45.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:46.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:46.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:47.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:47.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:48.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:48.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:49.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:49.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:50.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:50.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:51.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:51.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:52.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:52.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:53.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:53.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:54.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:54.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:55.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:55.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:56.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:56.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:57.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:57.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:58.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:58.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:45:59.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:45:59.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:00.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:00.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:00.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:46:00.832 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:46:00.832 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:46:00.832 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:46:00.832 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:46:00.852 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:46:00.852 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:46:00.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:00.853 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:01.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:01.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:01.850 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:01.881 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:02.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:02.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:03.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:03.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:04.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:04.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:04.881 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:04.881 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:05.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:05.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:05.888 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:05.920 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:06.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:06.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:07.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:07.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:08.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:08.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:08.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:08.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:09.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:09.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:10.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:10.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:11.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:11.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:12.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:12.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:13.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:13.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:13.784 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":384,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:13.806 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":384,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:14.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:14.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:15.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:15.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:16.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:16.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:16.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:16.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:17.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:17.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:17.777 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":384,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:17.800 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":384,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:18.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:18.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:19.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:19.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:20.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:20.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:20.801 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:20.801 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:21.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:21.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:21.784 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:21.806 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:21.806 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:46:21.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:21.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:46:22.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:22.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:22.755 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":388,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:22.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:22.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:46:23.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:23.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:23.768 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":388,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:23.792 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:23.792 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:46:24.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:24.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:24.788 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":388,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:24.818 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:46:24.819 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:24.819 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:46:25.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:25.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:25.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:46:26.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:26.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:26.889 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:46:26.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:26.889 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:27.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:27.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:27.886 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:27.915 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:27.915 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:46:27.915 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:46:27.916 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:27.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:46:28.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:28.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:28.930 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":388,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:28.953 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:28.953 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:46:29.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:29.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:29.949 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":388,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:29.971 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:29.972 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:46:30.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:30.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:30.967 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":388,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:30.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:46:30.998 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:46:30.998 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:46:30.998 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:46:30.998 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:46:30.998 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:46:31.017 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:46:31.018 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:46:31.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:31.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:31.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:31.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:32.025 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:32.050 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":388,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:32.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:32.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:33.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:33.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:34.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:34.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:35.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:35.051 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:46:35.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:35.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:36.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:36.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:37.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:37.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:38.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:38.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:39.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:39.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:40.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:40.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:41.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:41.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:42.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:42.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:43.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:43.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:44.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:44.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:45.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:45.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:46.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:46.057 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:46:46.057 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:46:46.058 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:46.058 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:46:46.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:46.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:47.050 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":390,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:47.073 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:46:47.073 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:46:47.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:47.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:48.062 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":390,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:46:48.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:46:48.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:48.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:49.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:49.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:50.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:50.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:51.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:51.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:52.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:52.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:53.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:53.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:54.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:54.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:55.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:55.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:56.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:56.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:57.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:57.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:58.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:58.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:46:59.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:46:59.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:00.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:00.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:01.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:01.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:02.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:02.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:03.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:03.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:04.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:04.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:05.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:05.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:06.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:06.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:07.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:07.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:08.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:08.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:09.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:09.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:10.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:10.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:11.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:11.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:12.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:12.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:13.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:13.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:14.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:14.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:15.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:15.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:16.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:16.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:17.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:17.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:18.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:18.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:19.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:19.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:20.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:20.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:21.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:21.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:22.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:22.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:23.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:23.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:24.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:24.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:25.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:25.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:26.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:26.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:27.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:27.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:28.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:28.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:29.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:29.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:30.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:30.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:31.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:31.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:32.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:32.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:33.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:33.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:34.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:34.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:35.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:35.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:36.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:36.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:37.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:37.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:38.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:38.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:39.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:39.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:40.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:40.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:41.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:41.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:42.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:42.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:43.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:43.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:44.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:44.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:45.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:45.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:46.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:46.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:47.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:47.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:48.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:48.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:49.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:49.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:50.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:50.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:51.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:51.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:52.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:52.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:53.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:53.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:54.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:54.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:55.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:55.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:56.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:56.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:57.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:57.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:58.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:58.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:47:59.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:47:59.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:00.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:00.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:01.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:01.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:02.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:02.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:03.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:03.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:04.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:04.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:05.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:05.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:06.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:06.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:07.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:07.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:08.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:08.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:09.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:09.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:10.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:10.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:11.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:11.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:12.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:12.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:13.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:13.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:14.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:14.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:15.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:15.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:16.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:16.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:17.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:17.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:18.093 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:48:18.093 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:48:18.093 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:48:18.093 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:48:18.094 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:48:18.099 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:48:18.099 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:48:18.100 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:18.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:18.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:18.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:19.110 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:19.132 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:19.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:19.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:20.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:20.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:21.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:21.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:22.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:22.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:22.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:22.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:23.142 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:23.165 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:23.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:23.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:24.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:24.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:25.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:25.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:26.166 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:26.166 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:26.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:26.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:27.159 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:27.182 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:27.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:27.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:28.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:28.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:29.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:29.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:30.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:30.184 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:30.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:30.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:31.182 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:31.204 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":390,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:31.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:31.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:32.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:32.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:33.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:33.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:34.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:34.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:34.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:34.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:35.222 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:35.246 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:35.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:48:35.246 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:35.246 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:48:35.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:35.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:36.242 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":394,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:36.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:36.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:48:36.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:36.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:37.246 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":394,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:37.278 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:37.278 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:48:37.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:37.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:38.291 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":394,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:38.314 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:48:38.315 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:38.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:48:38.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:38.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:39.415 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:48:39.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:39.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:40.416 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:48:40.416 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:40.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:40.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:40.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:41.405 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:41.433 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:41.434 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:48:41.434 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:48:41.434 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:41.434 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:48:41.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:41.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:42.417 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":394,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:48:42.449 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:42.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:48:42.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:42.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:43.432 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":394,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:43.458 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:43.458 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:48:43.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:43.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:44.464 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":394,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:44.494 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:48:44.494 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:48:44.494 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:48:44.495 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:48:44.495 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:48:44.495 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:48:44.514 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:48:44.514 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:48:44.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:44.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:44.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:44.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:45.544 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:45.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:45.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:45.577 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:46.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:46.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:47.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:47.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:48.578 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:48.578 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:48.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:48.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:49.550 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:49.581 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:49.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:49.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:50.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:50.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:51.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:51.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:52.582 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:52.582 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:48:52.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:52.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:53.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:53.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:54.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:54.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:55.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:55.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:56.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:56.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:57.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:57.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:58.608 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:58.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:58.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:58.646 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:48:58.646 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:48:58.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:58.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:48:59.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:48:59.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:48:59.636 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":396,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:48:59.665 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:48:59.665 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:49:00.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:00.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:00.654 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":396,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:49:00.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:49:01.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:01.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:02.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:02.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:03.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:03.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:04.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:04.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:05.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:05.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:06.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:06.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:07.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:07.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:08.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:08.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:09.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:09.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:10.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:10.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:11.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:11.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:12.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:12.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:13.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:13.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:14.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:14.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:15.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:15.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:16.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:16.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:17.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:17.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:18.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:18.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:19.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:19.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:20.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:20.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:21.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:21.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:22.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:22.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:23.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:23.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:24.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:24.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:25.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:25.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:26.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:26.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:27.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:27.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:28.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:28.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:29.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:29.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:30.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:30.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:31.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:31.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:32.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:32.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:33.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:33.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:34.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:34.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:35.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:35.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:36.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:36.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:37.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:37.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:38.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:38.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:39.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:39.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:40.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:40.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:41.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:41.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:42.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:42.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:43.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:43.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:44.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:44.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:45.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:45.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:46.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:46.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:47.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:47.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:48.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:48.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:49.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:49.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:50.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:50.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:51.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:51.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:52.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:52.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:53.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:53.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:54.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:54.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:55.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:55.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:56.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:56.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:57.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:57.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:58.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:58.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:49:59.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:49:59.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:00.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:00.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:01.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:01.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:02.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:02.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:03.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:03.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:04.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:04.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:05.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:05.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:06.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:06.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:07.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:07.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:08.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:08.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:09.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:09.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:10.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:10.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:11.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:11.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:12.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:12.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:13.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:13.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:14.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:14.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:15.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:15.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:16.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:16.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:17.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:17.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:18.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:18.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:19.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:19.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:20.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:20.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:21.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:21.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:22.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:22.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:23.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:23.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:24.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:24.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:25.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:25.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:26.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:26.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:27.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:27.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:28.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:28.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:29.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:29.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:30.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:30.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:30.679 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:50:30.679 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:50:30.679 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:50:30.679 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:50:30.680 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:50:30.702 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:50:30.702 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:50:30.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:30.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:31.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:31.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:31.699 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:31.729 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:32.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:32.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:33.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:33.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:34.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:34.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:34.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:34.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:35.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:35.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:35.709 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:35.730 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:36.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:36.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:37.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:37.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:38.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:38.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:38.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:38.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:39.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:39.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:39.683 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:39.715 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:40.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:40.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:41.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:41.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:42.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:42.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:42.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:42.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:43.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:43.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:43.736 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:43.763 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":396,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:44.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:44.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:45.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:45.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:46.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:46.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:46.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:46.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:47.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:47.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:47.747 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:47.776 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:47.777 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:50:47.777 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:47.777 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:50:48.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:48.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:48.785 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":400,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:48.808 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:48.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:50:49.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:49.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:49.793 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":400,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:49.818 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:49.818 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:50:50.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:50.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:50.830 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":400,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:50.853 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:50:50.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:50.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:50:51.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:51.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:51.983 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:50:52.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:52.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:52.984 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:50:52.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:52.984 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:53.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:53.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:53.991 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:54.017 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:54.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:50:54.017 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:50:54.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:54.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:50:54.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:54.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:55.028 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":400,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:55.049 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:55.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:50:55.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:55.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:55.979 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":400,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:56.009 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:56.009 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:50:56.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:56.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:57.008 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":400,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:50:57.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:50:57.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:50:57.038 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:50:57.039 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:50:57.039 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:50:57.039 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:50:57.069 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:50:57.069 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:50:57.069 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:50:57.069 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:50:57.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:57.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:58.151 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:58.175 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":400,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:50:58.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:58.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:50:59.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:50:59.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:00.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:00.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:01.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:51:01.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:51:01.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:01.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:02.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:02.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:03.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:03.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:04.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:04.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:05.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:05.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:06.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:06.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:07.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:07.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:08.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:08.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:09.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:09.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:10.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:10.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:11.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:11.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:11.904 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":404,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:51:11.935 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":404,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:51:11.936 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:51:11.936 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:51:11.936 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:51:12.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:12.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:12.572 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T03:51:12.560+0000 7f92b1f17700 0 --2- 172.21.6.124:0/2622345866 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f92ac07fcd0 0x7f92ac0800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T03:51:13.206 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":404,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:51:13.229 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:51:13.229 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:51:13.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:13.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:14.260 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":404,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:51:14.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:51:14.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:14.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:15.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:15.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:16.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:16.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:17.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:17.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:18.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:18.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:19.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:19.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:20.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:20.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:21.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:21.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:22.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:22.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:23.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:23.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:24.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:24.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:25.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:25.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:26.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:26.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:27.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:27.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:28.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:28.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:29.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:29.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:30.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:30.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:31.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:31.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:32.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:32.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:33.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:33.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:34.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:34.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:35.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:35.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:36.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:36.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:37.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:37.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:38.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:38.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:39.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:39.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:40.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:40.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:41.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:41.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:42.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:42.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:43.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:43.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:44.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:44.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:45.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:45.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:46.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:46.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:47.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:47.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:48.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:48.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:49.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:49.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:50.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:50.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:51.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:51.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:52.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:52.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:53.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:53.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:54.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:54.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:55.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:55.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:56.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:56.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:57.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:57.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:58.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:58.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:51:59.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:51:59.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:00.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:00.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:01.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:01.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:02.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:02.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:03.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:03.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:04.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:04.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:05.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:05.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:06.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:06.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:07.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:07.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:08.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:08.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:09.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:09.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:10.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:10.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:11.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:11.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:12.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:12.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:13.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:13.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:14.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:14.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:15.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:15.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:16.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:16.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:17.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:17.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:18.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:18.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:19.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:19.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:20.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:20.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:21.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:21.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:22.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:22.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:23.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:23.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:24.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:24.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:25.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:25.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:26.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:26.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:27.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:27.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:28.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:28.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:29.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:29.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:30.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:30.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:31.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:31.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:32.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:32.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:33.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:33.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:34.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:34.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:35.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:35.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:36.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:36.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:37.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:37.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:38.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:38.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:39.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:39.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:40.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:40.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:41.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:41.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:42.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:42.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:43.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:43.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:44.263 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:52:44.263 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:52:44.263 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:52:44.263 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:44.263 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:52:44.269 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:52:44.269 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:52:44.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:44.270 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:52:44.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:44.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:45.259 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":404,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:52:45.280 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":404,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:52:45.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:45.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:46.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:46.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:47.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:47.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:48.281 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:48.281 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:52:48.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:48.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:49.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:49.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:50.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:50.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:51.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:51.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:52.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:52.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:53.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:53.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:54.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:54.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:55.493 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:52:55.525 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:52:55.525 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:52:55.525 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:55.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:52:55.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:55.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:56.550 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":412,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:52:56.572 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:56.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:52:56.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:56.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:57.581 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":412,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:52:57.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:57.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:52:57.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:57.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:58.615 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":412,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:52:58.645 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:52:58.645 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:52:58.646 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:52:58.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:58.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:52:59.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:52:59.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:52:59.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:00.755 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:53:00.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:00.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:53:00.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:00.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:01.772 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:01.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:01.805 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:53:01.805 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:53:01.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:01.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:53:01.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:01.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:02.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:02.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:02.800 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":412,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:53:02.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:02.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:53:03.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:03.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:03.831 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":412,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:53:03.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:03.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:53:04.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:04.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:04.874 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":412,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:53:04.895 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:53:04.896 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:53:04.896 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:53:04.896 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:53:04.897 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:53:04.897 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:53:04.919 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:53:04.919 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:53:04.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:04.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:53:05.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:05.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:05.988 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:06.011 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:06.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:06.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:07.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:07.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:08.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:08.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:09.012 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:09.012 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:53:09.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:09.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:10.020 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:10.043 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:10.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:10.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:11.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:11.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:12.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:12.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:13.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:13.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:53:13.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:13.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:14.021 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:14.049 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:14.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:14.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:15.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:15.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:16.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:16.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:17.049 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:17.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:53:17.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:17.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:18.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:18.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:19.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:19.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:20.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:20.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:21.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":416,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:21.103 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":416,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:53:21.104 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:53:21.104 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:21.104 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:53:21.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:21.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:22.044 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":416,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:53:22.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:53:22.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:53:22.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:22.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:23.008 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":416,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:53:23.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:53:23.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:23.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:24.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:24.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:25.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:25.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:26.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:26.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:27.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:27.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:28.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:28.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:29.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:29.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:30.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:30.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:31.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:31.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:32.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:32.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:33.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:33.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:34.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:34.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:35.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:35.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:36.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:36.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:37.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:37.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:38.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:38.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:39.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:39.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:40.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:40.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:41.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:41.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:42.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:42.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:43.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:43.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:44.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:44.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:45.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:45.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:46.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:46.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:47.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:47.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:48.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:48.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:49.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:49.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:50.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:50.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:51.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:51.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:52.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:52.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:53.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:53.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:54.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:54.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:55.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:55.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:56.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:56.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:57.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:57.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:58.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:58.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:53:59.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:53:59.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:00.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:00.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:01.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:01.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:02.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:02.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:03.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:03.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:04.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:04.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:05.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:05.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:06.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:06.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:07.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:07.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:08.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:08.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:09.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:09.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:10.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:10.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:11.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:11.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:12.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:12.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:13.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:13.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:14.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:14.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:15.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:15.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:16.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:16.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:17.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:17.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:18.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:18.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:19.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:19.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:20.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:20.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:21.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:21.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:22.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:22.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:23.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:23.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:24.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:24.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:25.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:25.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:26.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:26.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:27.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:27.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:28.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:28.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:29.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:29.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:30.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:30.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:31.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:31.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:32.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:32.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:33.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:33.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:34.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:34.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:35.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:35.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:36.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:36.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:37.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:37.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:38.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:38.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:39.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:39.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:40.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:40.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:41.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:41.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:42.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:42.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:43.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:43.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:44.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:44.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:45.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:45.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:46.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:46.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:47.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:47.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:48.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:48.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:49.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:49.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:50.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:50.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:51.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:51.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:52.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:52.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:53.030 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:54:53.031 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T03:54:53.031 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T03:54:53.031 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:54:53.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T03:54:53.038 INFO:tasks.ceph.mon.c:Started 2019-10-01T03:54:53.038 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:54:53.039 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:54:53.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:54:53.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:53.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:54.026 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":416,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:54:54.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":416,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:54:54.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:54.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:55.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:55.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:56.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:56.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:57.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:54:57.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:54:57.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:57.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:58.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:58.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:54:59.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:54:59.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:00.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:00.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:01.167 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:01.187 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:01.187 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:55:01.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:01.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:55:01.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:01.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:02.194 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":422,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:55:02.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:02.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:55:02.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:02.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:03.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:03.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:04.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:04.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:04.672 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":422,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:04.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:04.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:55:05.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:05.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:05.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":422,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:05.716 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:55:05.716 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:05.716 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:55:06.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:06.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:06.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:55:07.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:07.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:07.788 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:55:07.789 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:07.789 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:55:08.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:08.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:08.859 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:08.880 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:08.880 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:55:08.880 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:55:08.880 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:08.880 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:55:09.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:09.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:09.848 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":422,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:55:09.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:09.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:55:10.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:10.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:10.857 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":422,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:10.883 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:10.883 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:55:11.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:11.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:11.854 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":422,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:11.884 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T03:55:11.885 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:55:11.885 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T03:55:11.885 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:55:11.885 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T03:55:11.886 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:55:11.910 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T03:55:11.910 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:55:11.911 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:11.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:55:12.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:12.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:12.906 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:12.928 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:13.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:13.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:14.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:14.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:15.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:15.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:15.929 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:15.930 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:55:16.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:16.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:16.942 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:16.964 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:17.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:17.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:18.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:18.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:19.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:19.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:19.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:19.967 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:55:20.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:20.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:20.966 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:20.997 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":422,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:21.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:21.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:22.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:22.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:23.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:23.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:23.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:23.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:55:24.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:24.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:25.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:25.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:26.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:26.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:27.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:27.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:28.055 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:28.076 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:55:28.077 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:55:28.077 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:28.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:55:28.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:28.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:29.093 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":424,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:29.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:55:29.117 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:55:29.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:29.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:30.137 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":424,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:55:30.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:55:30.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:30.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:31.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:31.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:32.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:32.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:33.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:33.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:34.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:34.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:35.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:35.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:36.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:36.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:37.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:37.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:38.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:38.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:39.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:39.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:40.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:40.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:41.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:41.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:42.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:42.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:43.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:43.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:44.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:44.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:45.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:45.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:46.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:46.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:47.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:47.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:48.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:48.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:49.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:49.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:50.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:50.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:51.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:51.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:52.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:52.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:53.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:53.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:54.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:54.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:55.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:55.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:56.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:56.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:57.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:57.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:58.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:58.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:55:59.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:55:59.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:00.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:00.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:01.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:01.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:02.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:02.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:03.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:03.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:04.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:04.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:05.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:05.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:06.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:06.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:07.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:07.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:08.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:08.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:09.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:09.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:10.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:10.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:11.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:11.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:12.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:12.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:13.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:13.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:14.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:14.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:15.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:15.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:16.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:16.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:17.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:17.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:18.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:18.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:19.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:19.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:20.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:20.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:21.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:21.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:22.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:22.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:23.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:23.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:24.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:24.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:25.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:25.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:26.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:26.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:27.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:27.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:28.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:28.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:29.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:29.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:30.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:30.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:31.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:31.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:32.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:32.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:33.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:33.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:34.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:34.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:35.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:35.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:36.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:36.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:37.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:37.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:38.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:38.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:39.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:39.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:40.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:40.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:41.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:41.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:42.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:42.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:43.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:43.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:44.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:44.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:45.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:45.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:46.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:46.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:47.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:47.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:48.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:48.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:49.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:49.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:50.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:50.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:51.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:51.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:52.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:52.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:53.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:53.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:54.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:54.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:55.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:55.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:56.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:56.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:57.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:57.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:58.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:58.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:56:59.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:56:59.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:00.165 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T03:57:00.165 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T03:57:00.165 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T03:57:00.165 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T03:57:00.165 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T03:57:00.171 INFO:tasks.ceph.mon.b:Started 2019-10-01T03:57:00.171 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:57:00.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:00.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:00.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:00.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:01.195 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:01.225 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:01.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:01.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:02.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:02.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:03.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:03.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:04.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:04.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:04.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:04.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:05.198 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:05.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":424,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:05.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:05.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:06.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:06.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:07.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:07.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:08.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:08.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:08.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:08.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:09.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:09.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:10.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:10.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:11.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:11.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:12.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:12.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:13.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:13.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:13.760 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:13.782 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:13.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:57:13.782 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:13.782 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:57:14.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:14.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:14.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":432,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:14.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:14.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:57:15.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:15.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:15.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":432,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:15.827 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:15.827 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:57:16.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:16.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:16.835 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":432,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:16.857 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:57:16.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:16.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:57:17.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:17.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:17.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:57:18.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:18.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:18.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:57:18.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:18.979 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:19.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:19.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:19.955 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:19.985 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:19.985 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:57:19.985 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:57:19.985 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:19.985 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:57:20.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:20.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:20.997 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":432,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:21.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:21.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:57:21.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:21.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:22.029 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":432,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:22.062 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:22.062 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:57:22.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:22.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:23.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":432,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:23.053 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T03:57:23.053 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:57:23.053 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T03:57:23.054 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:57:23.054 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T03:57:23.054 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:57:23.078 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T03:57:23.078 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:57:23.078 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:23.079 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:23.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:23.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:24.141 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:24.170 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:24.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:24.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:25.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:25.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:26.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:26.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:27.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:27.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:27.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:27.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:28.146 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:28.177 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:28.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:28.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:29.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:29.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:30.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:30.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:31.178 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:31.178 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:31.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:31.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:32.186 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:32.215 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":432,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:32.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:32.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:33.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:33.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:34.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:34.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:35.216 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:35.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:57:35.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:35.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:36.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:36.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:37.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:37.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:38.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:38.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:38.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:38.297 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:57:38.297 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:57:38.298 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:38.298 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:57:39.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:39.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:39.374 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":436,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:39.406 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:57:39.406 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:57:40.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:40.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:40.426 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":436,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:57:40.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:57:41.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:41.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:42.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:42.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:43.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:43.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:44.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:44.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:45.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:45.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:46.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:46.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:47.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:47.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:48.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:48.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:49.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:49.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:50.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:50.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:51.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:51.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:52.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:52.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:53.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:53.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:54.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:54.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:55.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:55.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:56.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:56.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:57.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:57.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:58.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:58.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:57:59.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:57:59.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:00.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:00.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:01.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:01.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:02.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:02.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:03.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:03.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:04.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:04.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:05.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:05.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:06.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:06.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:07.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:07.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:08.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:08.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:09.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:09.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:10.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:10.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:11.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:11.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:12.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:12.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:13.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:13.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:14.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:14.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:15.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:15.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:16.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:16.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:17.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:17.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:18.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:18.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:19.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:19.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:20.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:20.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:21.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:21.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:22.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:22.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:23.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:23.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:24.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:24.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:25.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:25.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:26.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:26.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:27.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:27.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:28.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:28.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:29.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:29.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:30.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:30.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:31.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:31.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:32.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:32.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:33.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:33.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:34.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:34.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:35.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:35.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:36.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:36.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:37.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:37.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:38.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:38.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:39.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:39.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:40.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:40.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:41.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:41.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:42.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:42.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:43.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:43.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:44.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:44.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:45.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:45.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:46.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:46.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:47.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:47.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:48.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:48.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:49.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:49.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:50.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:50.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:51.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:51.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:52.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:52.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:53.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:53.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:54.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:54.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:55.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:55.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:56.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:56.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:57.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:57.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:58.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:58.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:58:59.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:58:59.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:01.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:01.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:02.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:02.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:03.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:03.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:04.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:04.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:05.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:05.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:06.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:06.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:07.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:07.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:08.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:08.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:09.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:09.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:10.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:10.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:10.450 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T03:59:10.450 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T03:59:10.450 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T03:59:10.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:10.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T03:59:10.459 INFO:tasks.ceph.mon.a:Started 2019-10-01T03:59:10.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:59:10.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:10.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:11.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:11.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:11.485 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:11.515 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:12.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:12.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:13.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:13.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:14.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:14.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:14.516 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:14.516 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:15.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:15.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:15.530 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:15.553 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:15.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:15.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:16.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:16.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:17.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:17.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:18.554 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:18.554 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:18.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:18.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:19.616 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:19.641 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":436,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:19.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:19.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:20.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:20.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:21.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:21.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:22.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:22.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:22.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:22.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:23.633 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:23.664 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:23.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:59:23.665 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:23.665 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:59:23.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:23.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:24.612 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":438,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:24.644 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:24.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:59:24.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:24.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:25.700 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":438,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:25.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:25.730 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:59:25.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:25.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:26.717 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":438,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:26.740 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T03:59:26.740 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:26.740 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T03:59:26.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:26.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:27.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:27.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:27.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T03:59:28.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:28.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:28.867 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T03:59:28.868 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:28.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:29.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:29.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:29.851 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:29.874 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:29.874 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T03:59:29.875 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T03:59:29.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:29.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:59:30.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:30.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:30.900 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":438,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:30.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:30.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T03:59:31.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:31.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:31.920 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":438,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:31.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:31.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:59:32.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:32.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:32.943 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":438,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:32.965 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T03:59:32.966 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T03:59:32.966 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T03:59:32.966 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T03:59:32.966 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T03:59:32.966 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T03:59:32.988 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T03:59:32.988 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T03:59:32.988 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:32.988 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:33.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:33.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:34.006 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:34.028 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:34.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:34.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:35.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:35.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:37.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:37.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:37.029 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:37.029 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:38.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:38.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:38.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:38.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:39.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:39.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:40.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:40.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:41.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:41.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:41.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:41.060 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:42.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:42.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:42.083 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:42.091 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:42.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:42.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:44.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:44.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:45.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:45.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:45.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:45.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T03:59:45.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:45.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:47.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:47.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:48.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:48.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:48.266 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":442,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:48.298 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":442,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T03:59:48.298 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T03:59:48.298 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:48.299 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T03:59:48.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:48.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:49.309 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":442,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T03:59:49.334 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T03:59:49.334 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T03:59:49.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:49.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:50.301 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":442,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T03:59:50.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T03:59:50.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:50.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:51.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:51.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:53.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:53.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:54.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:54.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:55.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:55.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:56.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:56.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:57.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:57.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:58.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:58.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:58.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:58.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T03:59:59.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T03:59:59.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:00.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:00.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:01.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:01.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:02.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:02.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:04.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:04.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:04.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:04.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:06.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:06.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:07.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:07.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:08.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:08.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:09.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:09.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:10.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:10.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:11.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:11.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:12.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:12.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:13.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:13.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:14.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:14.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:15.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:15.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:16.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:16.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:17.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:17.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:18.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:18.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:19.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:19.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:20.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:20.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:21.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:21.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:22.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:22.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:23.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:23.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:24.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:24.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:25.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:25.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:26.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:26.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:27.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:27.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:28.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:28.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:29.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:29.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:30.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:30.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:31.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:31.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:32.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:32.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:33.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:33.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:34.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:34.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:35.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:35.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:36.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:36.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:37.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:37.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:38.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:38.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:39.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:39.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:39.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:39.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:41.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:41.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:41.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:41.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:42.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:42.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:43.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:43.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:44.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:44.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:45.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:45.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:47.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:47.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:47.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:47.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:48.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:48.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:50.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:49.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:50.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:50.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:51.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:51.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:52.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:52.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:53.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:53.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:54.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:54.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:55.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:55.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:56.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:56.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:57.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:57.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:58.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:58.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:00:59.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:00:59.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:00.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:00.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:02.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:02.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:02.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:02.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:03.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:03.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:04.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:04.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:05.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:05.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:06.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:06.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:07.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:07.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:08.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:08.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:09.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:09.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:10.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:10.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:11.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:11.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:12.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:12.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:13.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:13.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:14.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:14.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:15.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:15.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:16.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:16.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:17.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:17.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:18.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:18.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:19.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:19.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:20.332 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:01:20.332 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:01:20.332 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:01:20.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:20.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:01:20.359 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:01:20.359 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:01:20.359 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:20.359 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:20.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:20.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:21.365 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":442,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:21.400 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":442,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:21.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:21.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:22.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:22.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:23.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:23.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:24.401 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:24.401 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:24.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:24.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:25.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:25.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:26.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:26.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:27.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:27.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:28.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:28.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:29.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:29.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:30.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:30.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:31.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:31.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:32.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:32.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:33.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:33.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:34.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:34.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:35.433 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:35.465 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:35.465 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:01:35.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:35.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:01:35.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:35.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:36.467 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":452,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:01:36.499 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:36.499 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:01:36.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:36.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:37.503 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":452,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:01:37.526 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:37.527 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:01:37.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:37.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:38.530 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":452,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:01:38.552 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:01:38.553 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:38.553 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:01:38.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:38.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:39.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:01:39.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:39.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:40.650 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:01:40.650 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:40.650 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:40.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:40.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:41.648 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:41.679 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:41.679 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:01:41.680 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:01:41.680 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:41.680 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:01:41.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:41.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:42.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":452,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:01:42.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:42.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:01:42.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:42.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:43.787 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":452,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:01:43.808 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:43.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:01:43.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:43.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:44.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":452,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:01:44.820 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:01:44.820 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:01:44.820 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:01:44.820 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:01:44.821 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:01:44.821 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:01:44.848 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:01:44.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:01:44.848 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:44.848 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:44.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:44.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:45.861 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:45.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:45.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:45.886 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:46.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:46.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:47.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:47.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:48.887 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:48.888 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:48.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:48.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:49.871 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:49.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:49.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:49.898 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:50.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:50.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:51.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:51.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:52.899 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:52.899 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:52.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:52.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:53.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:53.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:53.932 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:53.964 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":452,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:01:54.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:54.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:55.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:55.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:56.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:56.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:56.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:01:56.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:01:57.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:57.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:58.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:58.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:01:59.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:01:59.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:00.303 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:02:00.334 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:02:00.335 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:02:00.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:02:00.335 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:02:00.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:00.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:01.352 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":454,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:02:01.384 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:02:01.385 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:02:01.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:01.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:02.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":454,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:02:02.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:02:02.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:02.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:03.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:03.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:04.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:04.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:05.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:05.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:06.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:06.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:07.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:07.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:08.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:08.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:09.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:09.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:10.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:10.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:11.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:11.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:12.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:12.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:13.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:13.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:14.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:14.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:15.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:15.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:16.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:16.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:17.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:17.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:18.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:18.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:19.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:19.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:20.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:20.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:21.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:21.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:22.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:22.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:23.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:23.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:24.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:24.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:25.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:25.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:26.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:26.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:27.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:27.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:28.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:28.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:29.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:29.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:30.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:30.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:31.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:31.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:32.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:32.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:33.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:33.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:34.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:34.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:35.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:35.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:36.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:36.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:37.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:37.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:38.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:38.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:39.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:39.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:40.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:40.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:41.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:41.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:42.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:42.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:43.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:43.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:44.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:44.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:45.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:45.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:46.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:46.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:47.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:47.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:48.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:48.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:49.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:49.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:50.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:50.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:51.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:51.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:52.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:52.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:53.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:53.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:54.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:54.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:55.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:55.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:56.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:56.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:57.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:57.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:58.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:58.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:02:59.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:02:59.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:00.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:00.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:01.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:01.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:02.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:02.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:03.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:03.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:04.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:04.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:05.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:05.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:06.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:06.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:07.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:07.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:08.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:08.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:09.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:09.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:10.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:10.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:11.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:11.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:12.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:12.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:13.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:13.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:14.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:14.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:15.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:15.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:16.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:16.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:17.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:17.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:18.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:18.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:19.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:19.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:20.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:20.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:21.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:21.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:22.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:22.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:23.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:23.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:24.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:24.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:25.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:25.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:26.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:26.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:27.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:27.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:28.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:28.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:29.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:29.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:30.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:30.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:31.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:31.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:32.438 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:03:32.438 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:03:32.438 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:03:32.439 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:03:32.439 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:03:32.445 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:03:32.445 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:03:32.445 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:32.445 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:32.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:32.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:33.468 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:33.491 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:33.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:33.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:34.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:34.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:35.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:35.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:36.492 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:36.492 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:36.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:36.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:37.488 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:37.519 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:37.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:37.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:38.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:38.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:39.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:39.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:40.520 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:40.520 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:40.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:40.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:41.522 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:41.545 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:41.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:41.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:42.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:42.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:43.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:43.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:44.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:44.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:44.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:44.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:45.550 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:45.573 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":454,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:45.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:45.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:46.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:46.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:47.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:47.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:48.574 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:48.574 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:48.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:48.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:49.585 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:49.614 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:49.614 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:03:49.614 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:49.614 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:03:49.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:49.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:50.599 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":458,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:03:50.629 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:50.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:03:50.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:50.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:51.641 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":458,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:03:51.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:51.673 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:03:51.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:51.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:52.638 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":458,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:03:52.661 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:03:52.661 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:52.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:03:52.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:52.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:53.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:53.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:53.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:03:54.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:54.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:54.817 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:03:54.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:54.817 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:55.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:55.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:55.814 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:55.836 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:55.837 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:03:55.837 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:03:55.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:55.837 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:03:56.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:56.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:56.814 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":458,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:03:56.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:56.837 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:03:57.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:57.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:57.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":458,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:03:57.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:57.817 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:03:58.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:58.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:58.814 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":458,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:03:58.837 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:03:58.837 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:03:58.837 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:03:58.837 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:03:58.838 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:03:58.838 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:03:58.860 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:03:58.860 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:03:58.860 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:03:58.861 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:03:59.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:03:59.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:03:59.917 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:03:59.955 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:04:00.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:00.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:01.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:01.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:02.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:02.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:02.956 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:04:02.956 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:04:03.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:03.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:03.969 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:04:03.992 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":458,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:04:04.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:04.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:05.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:05.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:06.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:06.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:06.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:04:06.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:04:07.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:07.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:08.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:08.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:09.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:09.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:10.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:10.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:11.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:11.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:12.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:12.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:13.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:13.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:13.625 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:04:13.643 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:04:13.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:04:13.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:04:13.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:04:14.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:14.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:14.663 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":462,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:04:14.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:04:14.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:04:15.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:15.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:15.664 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":462,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:04:15.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:04:16.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:16.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:17.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:17.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:18.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:18.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:19.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:19.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:20.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:20.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:21.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:21.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:22.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:22.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:23.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:23.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:24.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:24.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:25.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:25.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:26.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:26.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:27.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:27.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:28.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:28.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:29.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:29.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:30.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:30.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:31.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:31.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:32.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:32.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:33.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:33.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:34.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:34.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:35.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:35.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:36.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:36.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:37.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:37.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:38.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:38.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:39.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:39.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:40.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:40.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:41.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:41.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:42.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:42.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:43.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:43.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:44.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:44.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:45.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:45.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:46.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:46.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:47.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:47.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:48.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:48.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:49.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:49.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:50.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:50.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:51.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:51.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:52.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:52.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:53.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:53.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:54.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:54.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:55.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:55.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:56.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:56.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:57.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:57.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:58.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:58.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:04:59.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:04:59.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:00.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:00.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:01.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:01.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:02.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:02.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:03.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:03.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:04.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:04.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:05.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:05.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:06.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:06.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:07.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:07.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:08.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:08.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:09.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:09.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:10.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:10.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:11.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:11.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:12.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:12.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:13.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:13.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:14.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:14.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:15.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:15.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:16.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:16.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:17.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:17.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:18.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:18.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:19.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:19.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:20.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:20.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:21.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:21.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:22.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:22.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:23.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:23.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:24.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:24.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:25.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:25.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:26.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:26.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:27.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:27.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:28.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:28.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:29.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:29.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:30.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:30.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:31.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:31.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:32.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:32.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:33.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:33.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:34.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:34.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:35.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:35.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:36.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:36.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:37.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:37.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:38.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:38.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:39.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:39.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:40.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:40.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:41.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:41.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:42.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:42.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:43.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:43.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:44.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:44.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:45.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:45.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:45.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:05:45.688 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:05:45.688 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:05:45.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:45.688 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:05:45.743 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:05:45.743 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:05:45.743 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:45.743 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:05:46.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:46.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:46.745 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:46.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:47.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:47.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:48.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:48.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:49.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:49.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:49.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:49.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:05:50.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:50.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:50.770 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:50.802 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:51.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:51.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:52.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:52.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:53.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:53.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:53.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:53.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:05:54.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:54.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:54.875 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:54.893 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":462,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:55.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:55.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:56.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:56.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:57.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:57.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:57.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:57.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:05:58.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:58.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:58.909 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:58.941 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:05:58.968 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:05:58.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:05:58.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:05:59.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:05:59.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:05:59.992 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":464,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:00.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:00.023 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:06:00.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:00.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:00.976 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":464,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:01.004 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:01.004 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:06:01.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:01.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:02.013 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":464,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:06:02.034 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:06:02.034 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:02.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:06:02.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:02.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:03.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:06:03.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:03.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:04.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:04.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:04.165 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:06:04.165 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:04.165 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:06:05.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:05.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:05.165 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:05.189 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:05.189 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:06:05.189 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:06:05.190 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:05.190 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:06:06.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:06.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:06.187 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":464,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:06.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:06.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:06:07.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:07.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:07.313 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":464,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:07.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:07.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:06:08.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:08.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:08.355 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":464,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:06:08.378 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:06:08.378 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:06:08.378 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:06:08.378 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:06:08.379 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:06:08.379 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:06:08.400 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:06:08.401 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:06:08.401 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:08.401 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:06:09.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:09.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:09.460 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:09.490 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:10.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:10.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:11.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:10.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:12.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:12.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:12.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:12.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:06:12.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:12.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:13.503 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:13.526 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:14.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:14.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:14.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:14.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:15.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:15.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:16.527 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:16.528 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:06:16.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:16.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:17.547 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:17.570 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:17.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:17.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:18.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:18.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:19.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:19.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:20.571 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:20.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:06:20.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:20.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:22.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:22.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:23.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:22.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:23.606 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":468,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:23.636 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":468,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:06:23.637 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:06:23.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:23.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:06:24.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:24.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:24.630 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":468,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:24.661 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:06:24.661 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:06:25.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:25.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:25.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":468,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:06:25.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:06:25.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:25.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:26.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:26.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:28.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:28.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:29.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:29.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:30.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:30.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:30.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:30.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:31.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:31.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:33.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:33.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:34.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:34.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:35.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:35.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:36.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:36.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:37.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:37.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:37.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:37.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:38.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:38.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:39.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:39.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:40.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:40.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:41.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:41.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:42.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:42.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:43.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:43.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:45.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:45.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:46.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:46.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:47.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:47.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:48.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:48.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:49.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:49.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:50.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:50.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:51.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:51.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:52.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:52.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:53.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:53.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:54.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:54.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:55.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:55.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:56.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:56.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:57.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:57.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:58.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:58.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:06:59.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:06:59.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:00.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:00.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:01.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:01.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:02.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:02.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:03.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:03.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:04.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:04.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:05.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:05.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:06.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:06.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:07.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:07.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:08.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:08.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:09.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:09.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:10.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:10.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:11.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:11.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:12.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:12.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:13.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:13.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:14.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:14.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:15.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:15.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:16.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:16.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:17.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:17.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:18.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:18.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:19.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:19.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:20.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:20.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:21.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:21.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:22.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:22.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:23.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:23.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:24.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:24.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:25.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:25.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:26.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:26.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:27.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:27.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:28.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:28.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:29.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:29.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:30.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:30.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:31.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:31.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:32.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:32.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:33.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:33.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:34.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:34.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:35.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:35.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:36.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:36.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:37.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:37.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:38.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:38.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:39.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:39.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:40.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:40.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:41.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:41.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:42.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:42.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:43.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:43.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:44.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:44.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:45.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:45.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:46.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:46.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:47.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:47.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:48.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:48.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:49.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:49.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:50.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:50.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:51.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:51.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:52.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:52.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:53.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:53.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:54.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:54.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:55.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:55.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:55.681 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:07:55.682 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:07:55.682 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:07:55.682 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:07:55.682 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:07:55.688 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:07:55.688 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:07:55.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:07:55.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:07:56.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:56.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:56.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":468,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:07:56.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":468,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:07:57.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:57.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:58.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:58.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:59.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:07:59.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:07:59.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:07:59.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:00.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:00.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:01.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:01.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:02.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:02.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:03.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:03.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:04.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:04.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:04.644 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:04.667 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:04.667 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:08:04.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:04.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:08:05.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:05.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:05.683 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":474,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:05.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:05.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:08:06.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:06.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:06.972 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":474,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:06.994 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:06.994 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:08:07.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:07.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:08.028 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":474,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:08.047 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:08:08.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:08.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:08:08.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:08.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:09.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:08:09.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:09.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:10.108 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:08:10.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:10.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:10.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:10.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:11.119 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:11.140 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:11.140 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:08:11.140 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:08:11.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:11.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:08:11.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:11.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:12.152 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":474,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:12.182 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:12.183 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:08:12.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:12.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:13.172 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":474,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:13.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:13.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:08:13.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:13.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:14.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":474,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:14.241 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:08:14.241 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:08:14.241 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:08:14.241 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:08:14.242 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:08:14.242 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:08:14.259 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:08:14.260 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:08:14.260 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:14.260 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:14.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:14.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:15.278 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:15.307 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:15.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:15.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:16.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:16.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:17.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:17.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:18.308 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:18.308 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:18.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:18.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:19.307 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:19.333 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:19.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:19.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:20.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:20.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:21.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:21.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:22.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:22.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:22.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:22.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:23.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:23.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:23.345 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:23.367 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":474,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:24.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:24.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:25.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:25.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:26.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:26.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:26.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:26.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:08:27.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:27.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:28.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:28.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:29.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:29.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:29.613 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":478,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:29.632 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":478,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:08:29.632 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:08:29.632 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:29.633 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:08:30.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:30.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:30.633 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":478,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:30.655 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:08:30.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:08:31.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:31.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:31.639 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":478,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:08:31.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:08:32.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:32.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:33.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:33.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:34.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:34.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:35.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:35.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:36.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:36.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:37.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:37.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:38.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:38.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:39.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:39.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:40.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:40.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:41.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:41.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:42.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:42.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:43.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:43.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:44.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:44.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:45.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:45.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:46.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:46.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:47.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:47.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:48.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:48.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:49.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:49.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:50.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:50.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:51.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:51.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:52.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:52.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:53.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:53.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:54.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:54.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:55.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:55.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:56.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:56.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:57.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:57.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:58.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:58.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:08:59.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:08:59.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:00.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:00.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:01.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:01.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:02.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:02.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:03.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:03.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:04.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:04.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:05.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:05.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:06.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:06.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:07.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:07.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:08.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:08.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:09.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:09.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:10.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:10.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:10.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:10.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:11.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:11.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:12.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:12.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:13.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:13.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:14.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:14.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:15.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:15.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:16.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:16.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:17.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:17.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:18.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:18.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:19.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:19.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:20.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:20.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:21.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:21.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:22.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:22.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:23.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:23.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:24.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:24.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:25.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:25.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:26.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:26.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:27.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:27.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:28.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:28.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:29.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:29.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:30.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:30.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:31.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:31.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:32.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:32.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:33.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:33.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:34.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:34.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:35.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:35.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:36.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:36.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:37.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:37.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:38.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:38.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:39.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:39.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:40.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:40.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:41.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:41.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:42.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:42.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:43.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:43.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:44.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:44.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:45.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:45.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:46.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:46.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:47.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:47.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:48.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:48.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:49.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:49.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:50.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:50.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:51.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:51.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:52.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:52.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:53.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:53.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:54.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:54.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:55.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:55.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:56.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:56.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:57.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:57.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:58.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:58.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:09:59.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:09:59.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:00.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:00.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:01.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:01.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:01.666 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:10:01.666 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:10:01.666 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:10:01.666 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:01.667 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:10:01.702 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:10:01.702 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:10:01.703 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:01.703 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:02.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:02.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:02.728 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":478,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:02.750 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":478,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:03.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:03.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:04.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:04.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:05.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:05.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:05.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:05.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:06.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:06.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:07.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:07.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:08.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:08.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:09.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:09.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:10.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:10.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:10.599 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:10.630 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:10.631 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:10:10.631 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:10.631 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:10:11.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:11.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:11.702 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":484,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:11.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:11.725 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:10:12.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:12.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:12.701 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":484,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:12.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:12.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:10:13.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:13.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:13.737 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":484,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:13.767 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:10:13.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:13.767 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:10:14.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:14.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:14.863 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:10:15.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:15.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:15.864 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:10:15.864 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:15.864 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:16.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:16.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:16.902 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:16.902 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:16.902 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:10:16.903 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:10:16.903 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:16.903 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:10:17.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:17.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:17.935 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":484,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:17.960 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:17.960 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:10:18.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:18.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:18.971 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":484,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:19.004 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:19.004 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:10:19.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:19.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:20.015 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":484,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:20.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:10:20.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:10:20.047 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:10:20.047 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:10:20.047 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:10:20.047 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:10:20.067 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:10:20.067 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:10:20.067 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:20.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:20.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:20.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:21.125 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:21.148 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:21.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:21.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:22.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:22.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:23.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:23.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:24.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:24.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:24.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:24.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:25.145 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:25.167 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":484,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:25.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:25.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:26.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:26.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:27.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:27.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:28.167 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:28.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:10:28.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:28.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:29.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:29.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:30.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:30.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:31.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:31.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:32.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:32.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:33.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:33.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:34.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:34.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:35.575 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":488,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:35.598 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":488,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:10:35.598 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:10:35.598 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:35.598 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:10:35.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:35.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:36.599 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":488,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:36.631 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:10:36.632 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:10:36.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:36.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:37.638 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":488,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:10:37.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:10:37.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:37.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:38.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:38.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:39.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:39.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:40.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:40.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:41.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:41.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:42.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:42.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:43.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:43.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:44.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:44.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:45.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:45.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:46.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:46.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:47.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:47.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:48.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:48.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:49.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:49.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:50.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:50.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:51.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:51.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:52.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:52.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:53.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:53.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:54.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:54.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:55.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:55.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:56.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:56.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:57.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:57.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:58.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:58.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:10:59.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:10:59.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:00.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:00.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:01.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:01.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:02.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:02.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:03.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:03.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:04.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:04.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:05.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:05.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:06.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:06.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:07.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:07.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:08.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:08.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:09.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:09.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:10.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:10.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:11.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:11.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:12.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:12.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:13.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:13.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:14.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:14.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:15.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:15.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:16.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:16.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:17.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:17.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:18.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:18.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:19.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:19.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:20.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:20.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:21.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:21.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:22.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:22.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:23.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:23.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:24.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:24.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:25.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:25.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:26.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:26.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:27.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:27.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:28.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:28.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:29.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:29.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:30.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:30.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:31.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:31.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:32.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:32.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:33.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:33.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:34.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:34.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:35.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:35.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:36.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:36.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:37.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:37.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:38.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:38.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:39.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:39.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:40.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:40.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:41.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:41.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:42.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:42.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:43.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:43.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:44.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:44.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:45.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:45.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:46.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:46.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:47.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:47.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:48.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:48.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:49.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:49.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:50.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:50.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:51.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:51.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:52.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:52.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:53.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:53.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:54.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:54.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:55.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:55.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:56.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:56.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:57.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:57.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:58.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:58.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:11:59.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:11:59.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:00.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:00.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:01.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:01.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:02.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:02.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:03.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:03.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:04.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:04.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:05.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:05.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:06.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:06.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:07.669 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:12:07.669 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:12:07.669 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:12:07.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:07.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:12:07.676 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:12:07.676 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:12:07.676 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:07.676 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:12:07.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:07.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:08.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":488,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:08.668 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":488,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:08.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:08.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:09.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:09.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:10.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:10.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:11.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:11.669 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:12:11.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:11.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:12.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:12.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:13.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:13.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:14.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:14.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:15.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:15.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:16.583 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":492,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:16.614 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":492,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:16.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:16.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:17.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:17.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:18.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:18.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:19.615 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:19.616 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:12:19.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:19.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:20.220 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T04:12:20.209+0000 7f0b04d05700 0 --2- 172.21.6.124:0/689594411 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f0b00083470 0x7f0b0008ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T04:12:20.848 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:20.870 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:20.870 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:12:20.871 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:20.871 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:12:20.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:20.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:21.879 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":494,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:21.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:21.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:12:21.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:21.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:22.882 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":494,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:22.911 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:22.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:12:22.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:22.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:23.971 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":494,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:23.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:23.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:23.992 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:12:23.992 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:23.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:12:25.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:24.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:25.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:12:26.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:26.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:26.091 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:12:26.091 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:26.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:12:27.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:27.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:27.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:27.078 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:27.079 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:12:27.079 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:12:27.079 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:27.079 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:12:28.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:28.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:28.069 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":494,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:28.094 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:28.094 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:12:29.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:29.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:29.077 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":494,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:29.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:29.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:12:30.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:30.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:30.091 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":494,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:30.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:12:30.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T04:12:30.317 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:12:30.317 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:12:30.317 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:12:30.318 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:12:30.357 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:12:30.357 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T04:12:30.358 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T04:12:30.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:12:31.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:31.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:31.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:31.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:32.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:32.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:34.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:34.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:35.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:35.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:36.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:36.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:36.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:36.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:37.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:37.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:38.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:38.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:39.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:39.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:40.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:40.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:41.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:41.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:42.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:42.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:43.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:43.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:44.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:44.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:45.358 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T04:12:45.359 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T04:12:45.359 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:12:45.359 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:45.359 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:12:45.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:45.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:46.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:46.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:47.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:47.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:48.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:48.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:49.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:49.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:50.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:50.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:51.021 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":498,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:51.043 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":498,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:12:51.044 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:12:51.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:51.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:12:51.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:51.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:52.034 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":498,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:52.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:12:52.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:12:52.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:52.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:53.067 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":498,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:12:53.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:12:53.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:53.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:54.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:54.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:55.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:55.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:56.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:56.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:57.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:57.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:58.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:58.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:12:59.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:12:59.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:00.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:00.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:01.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:01.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:02.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:02.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:03.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:03.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:04.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:04.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:05.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:05.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:06.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:06.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:07.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:07.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:08.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:08.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:10.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:10.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:11.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:11.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:12.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:12.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:13.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:13.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:14.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:14.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:15.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:15.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:16.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:16.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:17.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:17.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:18.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:18.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:19.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:19.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:20.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:20.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:21.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:21.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:22.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:22.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:23.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:23.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:24.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:24.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:25.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:25.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:26.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:26.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:26.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:26.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:28.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:28.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:28.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:28.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:29.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:29.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:30.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:30.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:32.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:32.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:33.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:33.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:33.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:33.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:35.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:35.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:36.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:36.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:37.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:37.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:38.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:38.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:39.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:39.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:40.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:40.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:41.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:41.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:42.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:42.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:43.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:43.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:44.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:44.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:45.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:45.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:46.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:46.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:47.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:47.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:48.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:48.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:49.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:49.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:50.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:50.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:51.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:51.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:52.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:52.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:53.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:53.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:54.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:54.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:55.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:55.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:56.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:56.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:57.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:57.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:58.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:58.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:13:59.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:13:59.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:00.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:00.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:01.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:01.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:02.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:02.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:03.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:03.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:04.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:04.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:05.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:05.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:06.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:06.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:07.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:07.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:08.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:08.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:09.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:09.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:10.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:10.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:11.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:11.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:12.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:12.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:13.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:13.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:14.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:14.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:15.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:15.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:16.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:16.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:17.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:17.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:18.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:18.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:19.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:19.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:20.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:20.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:21.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:21.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:22.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:22.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:23.091 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:14:23.091 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:14:23.091 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:14:23.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:23.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:14:23.099 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:14:23.099 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T04:14:23.100 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T04:14:23.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:14:23.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:23.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:24.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:24.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:25.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:25.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:26.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:26.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:27.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:27.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:28.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:28.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:29.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:29.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:30.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:30.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:31.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:31.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:32.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:32.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:33.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:33.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:34.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:34.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:35.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:35.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:36.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:36.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:37.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:37.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:38.101 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T04:14:38.101 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T04:14:38.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:14:38.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:38.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:14:38.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:38.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:39.077 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":502,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:14:39.109 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":502,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:14:39.109 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:14:39.110 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:39.110 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:14:39.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:39.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:39.711 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T04:14:39.708+0000 7f1b82d9d700 0 --2- 172.21.6.124:0/3034444722 >> v2:172.21.6.124:3300/0 conn(0x7f1b84149410 0x7f1b8407cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T04:14:40.347 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":502,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:40.369 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:40.369 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:14:40.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:40.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:41.369 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":502,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:41.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:41.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:41.401 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:41.401 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:14:42.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:42.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:42.475 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":502,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:42.505 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:14:42.505 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:42.506 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:14:43.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:43.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:43.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:14:44.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:44.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:44.612 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:14:44.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:44.613 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:14:45.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:45.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:45.596 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":502,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:14:45.619 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":502,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:14:45.619 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:14:45.619 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:14:45.620 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:45.620 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:14:46.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:46.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:46.647 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":502,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:46.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:46.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:14:47.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:47.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:47.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":502,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:47.740 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:14:47.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:14:48.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:48.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:48.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":502,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:14:48.741 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:14:48.741 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T04:14:48.742 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:14:48.742 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:14:48.742 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:14:48.742 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:14:48.757 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:14:48.757 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:14:48.757 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:14:48.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:14:49.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:49.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:50.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:50.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:51.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:51.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:52.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:52.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:53.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:53.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:54.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:54.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:55.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:55.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:56.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:56.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:57.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:57.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:58.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:58.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:14:59.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:14:59.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:00.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:00.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:01.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:01.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:02.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:02.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:03.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:03.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:03.758 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:15:03.759 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:15:03.759 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:15:03.760 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:15:03.760 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:15:04.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:04.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:05.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:05.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:06.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:06.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:07.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:07.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:08.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:08.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:09.407 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":504,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:15:09.437 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":504,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:15:09.438 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:15:09.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:15:09.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:15:09.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:09.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:10.425 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":504,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:15:10.455 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:15:10.455 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:15:10.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:10.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:11.461 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":504,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:15:11.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:15:11.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:11.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:12.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:12.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:13.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:13.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:14.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:14.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:15.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:15.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:16.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:16.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:17.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:17.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:18.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:18.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:19.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:19.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:20.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:20.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:21.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:21.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:22.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:22.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:23.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:23.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:24.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:24.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:25.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:25.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:26.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:26.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:27.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:27.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:28.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:28.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:29.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:29.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:30.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:30.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:31.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:31.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:32.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:32.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:33.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:33.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:34.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:34.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:35.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:35.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:36.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:36.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:37.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:37.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:38.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:38.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:39.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:39.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:40.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:40.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:41.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:41.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:42.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:42.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:43.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:43.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:44.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:44.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:45.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:45.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:46.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:46.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:47.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:47.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:48.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:48.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:49.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:49.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:50.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:50.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:51.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:51.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:52.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:52.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:53.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:53.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:54.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:54.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:55.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:55.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:56.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:56.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:57.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:57.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:58.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:58.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:15:59.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:15:59.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:00.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:00.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:01.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:01.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:02.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:02.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:03.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:03.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:04.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:04.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:05.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:05.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:06.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:06.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:07.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:07.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:08.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:08.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:09.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:09.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:10.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:10.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:11.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:11.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:12.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:12.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:13.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:13.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:14.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:14.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:15.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:15.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:16.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:16.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:17.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:17.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:18.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:18.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:19.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:19.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:20.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:20.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:21.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:21.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:22.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:22.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:23.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:23.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:24.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:24.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:25.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:25.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:26.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:26.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:27.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:27.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:28.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:28.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:29.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:29.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:30.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:30.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:31.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:31.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:32.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:32.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:33.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:33.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:34.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:34.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:35.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:35.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:36.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:36.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:37.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:37.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:38.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:38.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:39.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:39.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:40.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:40.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:41.492 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:16:41.492 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:16:41.492 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:16:41.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:16:41.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:16:41.499 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:16:41.499 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:16:41.499 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:16:41.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:16:41.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:41.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:42.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:42.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:43.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:43.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:44.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:44.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:45.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:45.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:46.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:46.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:47.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:47.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:48.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:48.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:49.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:49.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:50.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:50.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:51.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:51.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:52.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:52.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:53.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:53.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:54.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:54.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:55.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:55.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:56.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:56.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:56.500 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:16:56.501 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:16:56.501 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:16:56.502 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:16:56.502 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:16:57.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:57.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:57.535 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:16:57.566 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:16:57.566 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:16:57.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:16:57.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:16:58.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:58.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:58.625 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":508,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:16:58.656 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:16:58.656 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:16:59.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:16:59.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:16:59.649 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":508,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:16:59.676 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:16:59.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:17:00.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:00.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:00.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":508,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:00.633 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:17:00.634 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:00.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:17:01.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:01.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:01.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:17:02.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:02.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:02.675 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:17:02.675 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:02.675 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:17:03.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:03.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:03.672 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:03.698 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:03.699 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:17:03.699 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:17:03.699 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:03.699 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:17:04.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:04.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:04.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":508,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:04.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:04.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:17:05.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:05.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:05.740 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":508,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:05.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:05.767 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:17:06.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:06.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:06.805 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":508,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:06.828 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:17:06.828 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:17:06.828 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:17:06.828 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:17:06.829 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:17:06.829 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:17:06.847 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:17:06.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:17:06.848 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:06.848 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:17:07.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:07.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:07.933 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:07.955 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:08.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:08.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:09.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:09.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:10.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:10.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:10.956 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:10.956 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:17:11.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:11.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:11.949 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:11.967 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:12.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:12.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:13.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:13.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:14.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:14.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:14.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:14.969 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:17:15.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:15.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:15.949 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:15.979 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":508,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:16.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:16.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:17.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:17.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:18.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:18.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:18.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:18.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:17:19.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:19.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:20.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:20.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:21.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:21.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:22.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:22.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:23.028 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":512,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:23.051 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":512,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:17:23.051 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:17:23.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:23.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:17:23.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:23.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:24.051 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":512,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:24.069 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:17:24.069 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:17:24.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:24.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:25.078 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":512,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:17:25.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:17:25.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:25.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:26.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:26.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:27.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:27.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:28.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:28.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:29.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:29.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:30.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:30.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:31.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:31.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:32.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:32.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:33.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:33.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:34.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:34.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:35.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:35.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:36.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:36.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:37.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:37.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:38.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:38.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:39.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:39.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:40.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:40.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:41.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:41.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:42.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:42.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:43.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:43.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:44.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:44.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:45.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:45.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:46.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:46.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:47.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:47.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:48.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:48.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:49.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:49.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:50.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:50.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:51.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:51.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:52.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:52.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:53.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:53.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:54.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:54.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:55.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:55.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:56.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:56.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:57.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:57.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:58.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:58.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:17:59.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:17:59.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:00.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:00.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:01.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:01.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:02.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:02.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:03.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:03.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:04.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:04.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:05.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:05.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:06.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:06.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:07.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:07.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:08.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:08.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:09.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:09.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:10.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:10.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:11.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:11.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:12.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:12.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:13.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:13.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:14.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:14.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:15.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:15.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:16.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:16.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:17.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:17.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:18.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:18.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:19.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:19.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:20.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:20.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:21.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:21.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:22.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:22.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:23.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:23.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:24.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:24.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:25.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:25.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:26.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:26.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:27.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:27.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:28.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:28.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:29.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:29.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:30.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:30.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:31.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:31.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:32.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:32.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:33.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:33.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:34.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:34.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:35.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:35.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:36.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:36.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:37.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:37.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:38.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:38.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:39.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:39.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:40.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:40.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:41.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:41.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:42.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:42.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:43.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:43.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:44.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:44.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:45.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:45.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:46.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:46.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:47.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:47.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:48.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:48.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:49.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:49.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:50.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:50.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:51.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:51.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:52.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:52.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:53.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:53.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:54.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:54.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:55.101 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:18:55.101 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:18:55.101 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:18:55.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:18:55.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:18:55.107 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:18:55.107 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:18:55.108 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:18:55.108 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:18:55.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:55.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:56.090 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":512,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:18:56.122 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":512,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:18:56.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:56.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:57.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:57.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:58.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:58.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:18:59.123 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:18:59.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:18:59.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:18:59.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:00.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:00.155 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:00.155 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:19:00.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:00.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:19:00.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:00.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:01.128 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":514,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:01.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:01.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:19:01.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:01.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:02.100 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":514,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:02.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:02.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:19:02.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:02.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:03.108 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":514,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:03.130 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:19:03.130 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:03.130 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:19:03.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:03.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:04.215 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:19:04.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:04.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:05.216 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:19:05.216 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:05.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:19:05.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:05.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:06.218 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:06.248 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:06.248 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:19:06.248 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:19:06.248 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:06.249 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:19:06.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:06.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:07.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":514,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:07.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:07.202 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:19:07.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:07.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:08.152 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":514,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:08.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:08.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:19:08.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:08.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:09.152 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":514,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:09.174 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:19:09.174 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:19:09.174 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:19:09.175 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:19:09.175 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:19:09.175 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:19:09.214 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:19:09.214 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:19:09.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:09.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:19:09.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:09.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:10.228 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:10.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:10.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:10.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:11.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:11.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:12.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:12.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:13.260 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:13.260 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:19:13.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:13.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:14.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:14.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:15.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:15.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:16.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:16.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:17.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:17.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:18.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:18.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:19.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:19.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:20.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:20.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:21.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:21.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:22.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:22.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:23.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:23.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:24.280 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:24.311 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:19:24.312 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:19:24.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:24.312 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:19:24.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:24.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:25.311 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":516,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:19:25.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:19:25.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:19:25.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:25.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:26.358 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":516,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:19:26.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:19:26.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:26.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:27.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:27.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:28.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:28.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:29.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:29.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:30.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:30.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:31.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:31.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:32.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:32.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:33.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:33.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:34.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:34.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:35.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:35.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:36.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:36.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:37.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:37.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:38.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:38.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:39.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:39.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:40.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:40.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:41.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:41.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:42.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:42.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:43.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:43.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:44.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:44.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:45.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:45.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:46.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:46.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:47.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:47.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:48.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:48.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:49.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:49.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:50.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:50.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:51.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:51.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:52.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:52.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:53.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:53.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:54.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:54.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:55.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:55.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:56.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:56.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:57.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:57.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:58.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:58.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:19:59.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:19:59.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:00.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:00.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:01.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:01.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:02.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:02.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:03.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:03.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:04.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:04.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:05.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:05.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:06.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:06.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:07.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:07.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:08.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:08.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:09.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:09.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:10.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:10.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:11.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:11.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:12.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:12.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:13.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:13.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:14.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:14.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:15.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:15.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:16.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:16.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:17.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:17.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:18.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:18.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:19.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:19.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:20.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:20.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:21.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:21.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:22.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:22.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:23.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:23.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:24.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:24.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:25.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:25.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:26.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:26.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:27.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:27.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:28.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:28.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:29.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:29.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:30.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:30.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:31.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:31.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:32.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:32.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:33.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:33.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:34.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:34.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:35.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:35.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:36.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:36.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:37.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:37.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:38.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:38.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:39.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:39.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:40.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:40.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:41.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:41.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:42.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:42.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:43.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:43.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:44.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:44.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:45.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:45.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:46.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:46.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:47.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:47.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:48.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:48.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:49.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:49.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:50.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:50.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:51.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:51.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:52.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:52.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:53.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:53.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:54.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:54.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:55.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:55.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:56.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:56.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:56.389 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:20:56.390 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:20:56.390 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:20:56.390 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:20:56.390 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:20:56.396 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:20:56.396 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:20:56.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:20:56.396 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:20:57.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:57.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:57.396 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:20:57.418 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:20:58.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:58.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:20:59.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:20:59.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:00.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:00.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:00.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:00.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:01.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:01.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:01.436 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:01.457 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:02.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:02.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:03.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:03.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:04.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:04.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:04.458 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:04.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:05.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:05.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:05.470 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:05.493 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:06.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:06.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:07.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:07.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:08.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:08.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:08.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:08.495 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:09.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:09.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:09.501 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:09.531 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":516,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:10.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:10.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:11.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:11.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:12.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:12.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:12.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:12.533 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:13.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:13.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:13.487 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:13.517 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:13.517 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:21:13.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:13.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:21:14.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:14.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:14.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":520,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:21:14.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:14.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:21:15.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:15.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:15.562 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":520,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:21:15.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:15.593 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:21:16.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:16.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:16.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":520,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:21:16.633 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:21:16.634 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:16.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:21:17.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:17.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:17.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:21:18.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:18.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:18.693 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:21:18.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:18.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:19.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:19.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:19.711 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:19.733 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:19.733 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:21:19.733 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:21:19.733 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:19.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:21:20.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:20.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:20.712 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":520,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:21:20.735 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:20.735 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:21:21.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:21.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:21.742 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":520,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:21:21.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:21.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:21:22.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:22.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:22.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":520,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:21:22.768 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:21:22.768 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:21:22.768 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:21:22.768 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:21:22.769 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:21:22.769 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:21:22.792 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:21:22.793 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:21:22.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:22.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:23.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:23.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:23.828 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:23.850 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:24.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:24.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:25.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:25.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:26.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:26.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:26.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:26.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:27.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:27.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:27.832 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:27.854 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:28.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:28.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:29.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:29.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:30.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:30.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:30.855 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:30.855 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:31.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:31.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:31.861 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:31.892 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":520,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:32.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:32.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:33.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:33.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:34.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:34.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:34.893 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:34.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:21:35.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:35.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:36.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:36.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:37.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:37.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:37.994 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":524,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:38.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":524,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:21:38.013 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:21:38.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:38.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:21:38.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:38.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:39.025 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":524,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:21:39.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:21:39.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:21:39.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:39.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:40.028 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":524,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:21:40.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:21:40.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:40.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:41.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:41.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:42.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:42.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:43.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:43.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:44.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:44.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:45.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:45.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:46.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:46.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:47.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:47.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:48.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:48.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:49.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:49.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:50.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:50.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:51.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:51.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:52.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:52.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:53.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:53.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:54.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:54.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:55.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:55.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:56.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:56.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:57.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:57.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:58.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:58.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:21:59.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:21:59.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:00.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:00.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:01.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:01.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:02.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:02.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:03.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:03.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:04.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:04.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:05.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:05.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:06.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:06.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:07.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:07.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:08.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:08.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:09.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:09.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:10.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:10.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:11.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:11.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:12.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:12.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:13.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:13.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:14.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:14.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:15.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:15.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:16.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:16.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:17.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:17.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:18.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:18.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:19.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:19.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:20.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:20.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:21.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:21.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:22.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:22.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:23.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:23.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:24.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:24.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:25.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:25.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:26.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:26.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:27.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:27.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:28.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:28.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:29.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:29.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:30.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:30.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:31.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:31.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:32.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:32.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:33.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:33.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:34.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:34.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:35.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:35.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:36.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:36.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:37.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:37.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:38.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:38.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:39.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:39.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:40.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:40.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:41.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:41.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:42.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:42.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:43.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:43.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:44.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:44.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:45.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:45.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:46.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:46.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:47.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:47.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:48.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:48.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:49.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:49.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:50.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:50.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:51.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:51.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:52.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:52.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:53.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:53.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:54.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:54.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:55.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:55.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:56.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:56.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:57.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:57.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:58.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:58.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:22:59.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:22:59.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:00.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:00.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:01.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:01.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:02.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:02.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:03.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:03.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:04.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:04.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:05.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:05.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:06.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:06.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:07.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:07.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:08.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:08.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:09.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:09.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:10.058 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:23:10.059 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:23:10.059 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:23:10.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:10.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:23:10.065 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:23:10.066 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:23:10.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:10.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:10.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:10.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:11.026 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":524,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:11.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":524,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:11.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:11.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:12.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:12.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:13.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:13.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:14.058 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:14.058 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:14.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:14.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:15.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:15.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:16.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:16.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:17.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:17.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:18.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:18.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:18.953 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:18.983 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:18.984 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:23:18.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:18.984 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:23:19.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:19.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:19.993 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":530,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:20.025 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:20.026 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:23:20.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:20.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:21.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":530,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:21.053 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:21.053 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:23:21.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:21.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:22.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":530,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:23:22.101 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:23:22.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:22.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:23:22.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:22.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:23.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:23:23.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:23.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:24.239 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:23:24.239 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:24.239 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:24.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:24.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:25.248 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:25.278 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:25.278 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:23:25.278 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:23:25.279 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:25.279 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:23:25.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:25.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:26.287 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":530,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:26.310 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:26.311 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:23:26.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:26.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:27.309 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":530,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:27.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:27.329 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:23:27.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:27.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:28.239 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":530,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:23:28.260 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:23:28.260 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:23:28.260 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:23:28.261 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:23:28.261 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:23:28.261 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:23:28.281 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:23:28.281 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:23:28.282 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:28.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:28.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:28.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:29.290 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:29.321 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:29.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:29.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:30.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:30.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:31.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:31.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:32.322 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:32.322 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:32.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:32.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:33.323 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:33.344 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:33.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:33.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:34.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:34.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:35.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:35.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:36.345 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:36.346 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:36.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:36.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:37.339 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:37.360 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:37.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:37.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:38.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:38.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:39.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:39.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:40.361 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:40.361 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:23:40.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:40.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:41.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:41.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:42.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:42.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:43.510 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:43.531 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:23:43.532 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:23:43.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:43.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:23:43.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:43.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:44.532 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":532,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:44.563 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:23:44.563 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:23:44.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:44.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:45.558 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":532,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:23:45.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:23:45.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:45.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:46.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:46.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:47.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:47.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:48.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:48.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:49.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:49.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:50.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:50.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:51.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:51.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:52.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:52.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:53.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:53.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:54.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:54.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:55.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:55.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:56.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:56.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:57.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:57.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:58.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:58.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:23:59.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:23:59.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:00.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:00.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:01.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:01.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:02.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:02.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:03.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:03.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:04.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:04.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:05.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:05.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:06.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:06.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:07.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:07.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:08.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:08.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:09.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:09.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:10.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:10.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:11.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:11.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:12.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:12.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:13.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:13.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:14.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:14.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:15.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:15.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:16.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:16.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:17.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:17.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:18.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:18.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:19.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:19.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:20.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:20.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:21.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:21.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:22.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:22.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:23.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:23.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:24.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:24.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:25.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:25.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:26.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:26.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:27.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:27.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:28.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:28.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:29.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:29.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:30.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:30.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:31.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:31.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:32.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:32.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:33.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:33.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:34.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:34.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:35.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:35.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:36.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:36.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:37.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:37.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:38.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:38.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:39.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:39.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:40.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:40.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:41.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:41.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:42.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:42.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:43.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:43.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:44.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:44.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:45.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:45.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:46.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:46.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:47.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:47.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:48.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:48.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:49.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:49.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:50.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:50.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:51.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:51.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:52.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:52.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:53.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:53.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:54.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:54.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:55.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:55.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:56.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:56.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:57.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:57.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:58.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:58.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:24:59.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:24:59.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:00.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:00.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:01.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:01.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:02.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:02.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:03.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:03.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:04.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:04.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:05.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:05.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:06.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:06.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:07.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:07.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:08.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:08.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:09.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:09.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:10.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:10.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:11.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:11.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:12.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:12.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:13.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:13.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:14.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:14.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:15.584 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:25:15.584 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:25:15.585 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:25:15.585 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:25:15.585 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:25:15.629 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:25:15.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:25:15.630 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:15.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:15.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:15.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:16.664 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:16.694 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:16.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:16.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:17.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:17.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:18.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:18.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:19.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:19.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:19.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:19.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:20.724 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:20.749 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:20.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:20.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:21.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:21.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:22.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:22.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:23.750 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:23.751 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:23.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:23.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:24.759 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:24.785 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:24.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:24.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:25.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:25.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:26.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:26.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:27.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:27.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:28.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:28.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:28.780 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:28.812 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:29.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:29.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:30.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:30.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:31.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:31.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:31.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:31.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:32.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:32.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:32.839 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:32.870 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:32.870 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:25:32.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:32.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:25:33.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:33.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:33.864 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":536,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:33.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:33.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:25:34.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:34.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:34.902 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":536,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:34.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:34.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:25:35.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:35.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:35.946 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":536,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:35.985 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:25:35.985 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:35.985 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:25:36.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:36.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:37.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:25:37.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:37.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:38.092 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:25:38.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:38.093 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:38.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:38.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:39.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:39.091 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:39.091 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:25:39.092 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:25:39.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:39.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:25:39.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:39.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:40.073 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":536,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:40.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:40.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:25:40.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:40.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:41.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:41.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:41.315 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":536,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:41.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:41.339 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:25:42.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:42.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:42.345 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":536,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:42.367 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:25:42.367 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:25:42.367 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:25:42.367 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:25:42.368 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:25:42.368 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:25:42.387 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:25:42.387 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:25:42.388 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:42.388 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:43.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:43.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:43.375 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:43.404 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:44.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:44.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:45.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:45.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:46.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:46.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:46.405 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:46.405 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:47.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:47.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:47.394 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:47.415 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:48.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:48.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:49.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:49.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:50.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:50.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:50.416 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:50.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:51.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:51.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:51.432 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:51.453 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:52.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:52.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:53.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:53.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:54.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:54.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:54.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:54.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:25:55.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:55.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:56.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:56.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:57.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:57.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:57.937 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:57.967 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:25:57.968 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:25:57.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:57.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:25:58.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:58.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:58.960 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":538,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:58.992 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:25:58.992 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:25:59.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:25:59.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:25:59.934 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":538,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:25:59.955 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:26:00.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:00.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:01.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:01.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:02.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:02.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:03.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:03.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:04.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:04.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:05.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:05.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:06.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:06.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:07.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:07.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:08.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:08.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:09.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:09.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:10.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:10.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:11.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:11.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:12.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:12.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:13.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:13.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:14.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:14.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:15.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:15.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:16.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:16.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:17.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:17.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:18.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:18.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:19.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:19.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:20.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:20.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:21.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:21.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:22.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:22.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:23.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:23.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:24.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:24.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:25.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:25.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:26.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:26.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:27.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:27.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:28.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:28.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:29.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:29.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:30.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:30.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:31.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:31.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:32.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:32.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:33.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:33.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:34.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:34.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:35.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:35.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:36.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:36.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:37.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:37.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:38.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:38.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:39.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:39.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:40.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:40.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:41.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:41.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:42.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:42.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:43.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:43.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:44.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:44.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:45.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:45.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:46.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:46.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:47.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:47.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:48.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:48.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:49.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:49.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:50.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:50.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:51.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:51.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:52.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:52.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:53.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:53.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:54.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:54.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:55.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:55.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:56.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:56.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:57.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:57.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:58.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:58.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:26:59.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:26:59.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:00.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:00.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:01.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:01.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:02.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:02.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:03.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:03.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:04.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:04.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:05.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:05.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:06.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:06.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:07.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:07.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:08.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:08.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:09.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:09.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:10.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:10.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:11.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:11.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:12.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:12.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:13.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:13.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:14.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:14.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:15.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:15.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:16.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:16.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:17.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:17.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:18.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:18.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:19.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:19.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:20.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:20.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:21.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:21.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:22.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:22.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:23.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:23.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:24.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:24.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:25.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:25.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:26.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:26.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:27.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:27.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:28.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:28.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:29.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:29.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:29.956 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:27:29.956 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:27:29.956 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:27:29.956 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:27:29.956 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:27:29.964 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:27:29.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:27:29.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:29.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:30.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:30.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:30.962 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:30.989 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:31.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:31.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:32.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:32.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:33.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:33.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:33.990 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:33.990 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:34.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:34.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:34.979 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:35.010 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:35.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:35.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:36.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:36.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:37.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:37.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:38.010 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:38.010 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:38.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:38.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:38.992 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:39.014 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:39.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:39.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:40.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:40.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:41.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:41.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:42.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:42.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:42.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:42.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:43.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:43.045 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:43.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:43.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:44.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:44.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:45.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:45.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:46.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:46.046 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:46.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:46.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:47.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:47.082 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:47.082 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:27:47.082 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:47.083 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:27:47.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:47.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:48.083 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":542,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:48.107 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:48.107 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:27:48.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:48.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:49.103 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":542,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:49.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:49.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:27:49.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:49.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:50.130 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":542,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:50.157 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:27:50.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:50.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:27:50.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:50.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:51.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:51.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:51.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:27:52.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:52.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:52.216 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:27:52.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:52.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:53.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:53.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:53.216 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:53.247 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:53.247 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:27:53.247 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:27:53.248 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:53.248 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:27:54.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:54.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:54.249 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":542,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:54.281 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:54.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:27:55.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:55.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:55.276 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":542,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:55.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:55.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:27:56.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:56.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:56.323 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":542,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:27:56.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:27:56.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:27:56.342 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:27:56.342 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:27:56.342 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:27:56.342 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:27:56.364 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:27:56.365 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:27:56.365 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:27:56.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:27:57.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:57.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:57.435 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:57.460 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:27:58.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:58.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:27:59.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:27:59.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:00.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:00.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:00.461 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:28:00.461 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:28:01.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:01.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:01.477 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:28:01.501 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":542,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:28:02.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:02.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:03.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:03.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:04.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:04.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:04.502 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:28:04.503 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:28:05.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:05.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:06.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:06.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:07.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:07.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:08.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:08.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:09.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:09.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:10.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:10.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:11.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:11.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:11.956 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:28:11.978 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:28:11.978 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:28:11.979 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:28:11.979 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:28:12.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:12.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:12.958 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":546,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:28:12.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:28:12.982 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:28:13.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:13.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:13.977 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":546,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:28:13.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:28:14.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:14.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:15.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:15.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:16.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:16.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:17.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:17.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:18.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:18.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:19.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:19.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:20.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:20.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:21.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:21.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:22.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:22.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:23.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:23.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:24.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:24.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:25.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:25.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:26.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:26.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:27.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:27.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:28.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:28.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:29.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:29.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:30.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:30.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:31.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:31.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:32.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:32.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:33.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:33.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:34.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:34.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:35.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:35.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:36.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:36.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:37.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:37.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:38.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:38.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:39.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:39.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:40.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:40.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:41.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:41.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:42.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:42.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:43.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:43.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:44.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:44.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:45.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:45.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:46.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:46.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:46.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:46.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:48.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:48.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:48.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:48.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:49.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:49.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:50.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:50.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:51.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:51.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:52.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:52.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:53.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:53.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:55.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:55.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:55.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:55.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:56.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:56.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:57.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:57.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:58.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:58.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:28:59.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:28:59.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:00.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:00.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:01.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:01.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:02.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:02.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:03.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:03.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:04.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:04.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:05.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:05.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:06.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:06.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:07.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:07.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:08.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:08.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:09.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:09.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:10.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:10.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:11.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:11.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:12.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:12.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:13.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:13.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:14.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:14.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:15.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:15.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:16.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:16.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:18.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:18.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:19.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:19.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:20.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:20.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:21.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:21.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:22.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:22.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:23.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:23.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:24.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:24.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:25.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:25.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:26.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:26.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:27.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:27.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:28.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:28.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:29.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:29.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:30.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:30.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:31.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:31.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:32.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:32.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:33.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:33.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:34.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:34.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:35.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:35.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:36.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:36.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:37.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:37.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:38.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:38.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:39.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:39.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:40.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:40.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:41.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:41.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:42.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:42.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:43.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:43.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:44.000 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:29:44.000 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:29:44.000 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:29:44.000 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:44.001 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:29:44.006 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:29:44.006 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:29:44.006 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:44.006 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:29:44.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:44.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:44.978 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:45.000 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:45.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:45.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:46.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:46.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:47.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:47.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:48.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:48.002 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:29:48.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:48.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:49.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:49.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:50.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:50.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:51.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:51.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:52.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:52.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:52.940 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:52.965 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:52.965 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:29:52.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:52.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:29:53.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:53.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:53.965 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":552,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:29:53.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:53.997 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:29:54.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:54.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:54.996 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":552,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:29:55.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:55.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:29:55.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:55.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:56.024 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":552,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:29:56.054 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:29:56.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:56.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:29:56.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:56.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:57.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:29:57.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:57.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:58.182 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:29:58.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:58.183 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:29:58.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:58.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:59.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:29:59.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:29:59.419 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:59.441 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:29:59.441 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:29:59.441 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:29:59.441 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:29:59.442 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:30:00.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:00.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:00.419 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":552,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:30:00.448 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:00.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:30:01.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:01.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:01.431 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":552,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:30:01.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:01.455 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:30:02.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:02.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:02.478 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":552,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:30:02.504 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:30:02.504 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:30:02.504 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:30:02.504 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:30:02.505 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:30:02.505 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:30:02.523 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:30:02.523 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:30:02.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:02.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:30:03.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:03.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:03.589 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:03.611 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:04.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:04.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:05.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:05.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:06.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:06.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:06.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:06.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:30:07.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:07.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:07.614 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:07.637 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:08.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:08.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:09.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:09.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:10.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:10.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:10.638 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:10.639 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:30:11.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:11.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:11.652 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:11.676 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":552,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:12.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:12.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:13.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:13.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:14.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:14.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:14.680 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:14.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:30:15.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:15.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:16.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:16.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:17.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:17.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:18.018 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":556,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:18.050 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":556,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:30:18.051 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:30:18.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:18.051 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:30:18.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:18.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:19.082 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":556,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:30:19.113 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:30:19.113 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:30:19.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:19.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:20.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":556,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:30:20.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:30:20.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:20.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:21.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:21.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:22.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:22.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:23.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:23.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:24.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:24.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:25.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:25.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:26.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:26.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:27.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:27.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:28.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:28.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:29.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:29.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:30.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:30.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:31.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:31.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:32.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:32.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:33.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:33.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:34.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:34.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:35.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:35.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:36.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:36.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:37.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:37.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:38.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:38.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:39.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:39.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:40.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:40.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:41.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:41.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:42.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:42.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:43.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:43.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:44.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:44.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:45.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:45.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:46.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:46.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:47.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:47.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:48.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:48.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:49.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:49.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:50.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:50.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:51.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:51.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:52.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:52.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:53.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:53.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:54.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:54.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:55.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:55.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:56.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:56.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:57.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:57.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:58.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:58.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:30:59.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:30:59.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:00.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:00.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:01.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:01.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:02.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:02.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:03.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:03.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:04.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:04.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:05.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:05.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:06.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:06.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:07.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:07.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:08.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:08.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:09.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:09.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:10.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:10.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:11.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:11.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:12.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:12.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:13.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:13.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:14.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:14.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:15.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:15.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:16.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:16.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:17.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:17.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:18.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:18.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:19.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:19.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:20.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:20.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:21.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:21.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:22.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:22.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:23.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:23.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:24.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:24.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:25.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:25.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:26.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:26.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:27.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:27.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:28.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:28.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:29.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:29.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:30.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:30.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:31.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:31.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:32.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:32.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:33.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:33.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:34.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:34.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:35.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:35.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:36.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:36.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:37.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:37.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:38.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:38.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:39.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:39.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:40.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:40.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:41.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:41.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:42.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:42.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:43.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:43.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:44.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:44.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:45.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:45.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:46.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:46.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:47.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:47.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:48.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:48.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:49.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:49.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:50.156 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:31:50.157 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:31:50.157 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:31:50.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:31:50.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:31:50.164 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:31:50.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:31:50.165 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:31:50.165 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:31:50.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:50.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:51.181 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":556,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:31:51.212 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":556,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:31:51.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:51.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:52.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:52.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:53.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:53.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:54.213 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:31:54.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:31:54.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:54.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:55.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:55.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:56.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:56.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:57.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:57.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:58.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:58.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:31:58.972 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":562,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:31:58.994 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":562,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:31:58.995 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:31:58.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:31:58.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:31:59.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:31:59.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:00.011 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":562,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:32:00.090 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:00.090 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:32:00.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:00.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:01.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:01.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:01.721 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":562,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:01.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:01.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:32:02.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:02.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:02.664 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":562,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:02.686 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:32:02.686 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:02.687 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:32:03.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:03.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:03.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:32:04.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:04.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:04.799 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:32:04.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:04.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:32:05.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:05.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:05.849 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":562,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:32:05.879 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":562,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:32:05.879 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:32:05.879 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:32:05.879 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:05.879 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:32:06.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:06.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:06.889 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":562,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:32:06.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:06.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:32:07.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:07.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:07.820 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":562,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:07.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:07.851 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:32:08.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:08.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:08.842 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":562,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:08.864 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:32:08.864 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T04:32:08.864 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:32:08.865 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:32:08.865 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:32:08.865 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:32:08.910 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:32:08.910 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:32:08.910 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:32:08.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:32:09.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:09.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:10.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:10.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:11.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:11.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:12.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:12.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:13.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:13.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:14.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:14.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:15.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:15.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:16.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:16.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:17.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:17.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:18.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:18.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:19.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:19.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:20.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:20.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:21.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:21.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:22.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:22.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:23.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:23.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:23.911 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:32:23.911 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:32:23.912 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:32:23.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:23.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:32:24.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:24.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:25.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:25.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:26.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:26.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:27.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:27.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:28.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:28.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:29.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:29.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:30.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:30.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:31.112 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":564,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:32:31.135 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":564,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:32:31.135 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:32:31.135 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:31.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:32:31.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:31.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:32.129 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":564,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:32.152 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:32:32.152 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:32:32.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:32.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:33.149 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":564,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:32:33.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:33.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:33.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:32:34.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:34.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:35.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:35.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:36.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:36.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:37.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:37.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:38.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:38.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:39.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:39.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:40.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:40.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:41.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:41.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:42.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:42.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:43.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:43.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:44.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:44.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:45.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:45.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:45.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:45.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:47.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:47.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:47.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:47.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:49.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:49.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:49.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:49.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:50.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:50.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:51.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:51.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:52.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:52.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:54.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:54.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:55.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:55.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:56.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:56.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:57.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:57.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:57.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:57.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:58.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:58.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:32:59.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:32:59.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:00.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:00.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:01.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:01.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:02.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:02.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:03.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:03.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:04.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:04.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:05.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:05.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:06.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:06.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:07.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:07.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:08.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:08.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:09.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:09.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:10.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:10.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:11.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:11.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:12.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:12.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:13.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:13.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:14.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:14.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:15.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:15.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:17.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:17.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:18.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:18.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:19.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:19.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:20.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:20.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:21.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:21.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:22.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:22.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:23.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:23.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:23.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:23.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:24.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:24.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:25.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:25.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:26.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:26.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:27.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:27.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:28.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:28.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:29.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:29.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:30.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:30.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:31.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:31.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:32.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:32.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:33.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:33.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:34.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:34.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:35.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:35.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:36.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:36.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:37.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:37.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:38.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:38.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:40.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:40.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:41.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:41.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:42.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:42.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:43.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:43.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:44.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:44.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:45.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:45.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:46.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:46.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:47.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:47.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:48.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:48.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:49.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:49.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:50.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:50.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:51.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:51.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:52.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:52.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:53.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:53.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:54.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:54.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:55.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:55.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:56.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:56.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:57.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:57.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:58.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:58.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:33:59.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:33:59.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:00.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:00.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:01.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:01.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:02.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:02.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:03.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:03.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:03.174 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:34:03.174 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:34:03.174 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:34:03.174 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:03.174 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:34:03.193 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:34:03.193 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:34:03.193 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:34:03.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:34:04.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:04.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:05.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:05.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:06.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:06.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:07.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:07.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:08.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:08.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:09.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:09.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:10.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:10.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:11.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:11.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:12.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:12.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:13.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:13.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:14.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:14.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:15.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:15.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:16.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:16.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:17.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:17.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:18.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:18.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:18.194 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:34:18.195 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:34:18.195 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:34:18.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:18.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:34:19.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:19.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:19.191 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:19.221 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:19.221 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:34:19.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:19.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:34:20.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:20.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:20.228 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":568,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:20.250 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:20.250 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:34:21.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:21.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:21.246 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":568,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:21.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:21.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:34:22.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:22.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:22.207 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":568,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:22.230 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:34:22.230 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:22.230 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:34:23.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:23.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:23.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:34:24.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:24.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:24.295 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:34:24.296 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:24.296 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:34:25.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:25.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:25.309 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:25.340 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:25.340 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:34:25.340 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:34:25.340 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:25.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:34:26.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:26.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:26.336 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":568,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:26.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:26.353 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:34:27.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:27.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:27.342 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":568,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:27.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:27.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:34:28.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:28.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:28.362 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":568,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:28.393 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:34:28.393 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:34:28.394 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:34:28.394 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:34:28.394 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:34:28.394 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:34:28.409 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:34:28.410 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:34:28.410 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:28.410 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:34:29.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:29.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:29.448 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:29.478 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":568,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:29.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:29.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:30.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:30.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:31.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:31.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:32.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:32.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:34:32.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:32.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:33.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:33.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:34.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:34.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:35.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:35.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:36.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:36.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:37.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:37.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:38.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:38.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:39.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:39.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:40.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:40.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:41.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:41.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:42.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:42.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:43.105 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":572,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:43.127 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":572,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:34:43.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:34:43.127 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:43.127 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:34:43.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:43.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:44.140 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":572,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:44.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:34:44.162 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:34:44.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:44.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:45.128 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":572,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:34:45.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:34:45.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:45.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:46.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:46.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:47.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:47.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:48.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:48.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:49.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:49.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:50.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:50.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:51.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:51.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:52.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:52.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:53.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:53.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:54.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:54.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:55.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:55.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:56.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:56.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:57.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:57.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:58.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:58.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:34:59.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:34:59.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:00.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:00.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:01.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:01.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:02.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:02.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:03.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:03.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:04.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:04.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:05.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:05.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:06.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:06.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:07.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:07.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:08.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:08.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:09.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:09.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:10.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:10.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:11.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:11.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:12.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:12.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:13.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:13.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:14.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:14.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:15.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:15.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:16.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:16.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:17.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:17.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:18.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:18.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:19.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:19.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:20.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:20.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:21.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:21.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:22.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:22.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:23.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:23.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:24.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:24.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:25.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:25.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:26.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:26.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:27.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:27.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:28.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:28.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:29.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:29.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:30.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:30.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:31.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:31.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:32.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:32.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:33.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:33.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:34.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:34.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:35.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:35.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:36.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:36.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:37.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:37.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:38.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:38.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:39.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:39.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:40.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:40.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:41.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:41.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:42.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:42.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:43.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:43.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:44.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:44.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:45.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:45.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:46.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:46.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:47.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:47.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:48.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:48.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:49.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:49.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:50.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:50.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:51.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:51.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:52.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:52.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:53.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:53.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:54.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:54.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:56.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:56.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:57.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:57.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:58.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:58.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:35:59.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:35:59.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:00.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:00.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:01.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:01.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:02.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:02.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:03.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:03.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:04.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:04.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:05.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:05.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:06.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:06.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:07.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:07.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:08.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:08.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:09.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:09.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:10.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:10.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:11.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:11.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:12.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:12.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:13.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:13.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:14.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:14.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:15.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:15.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:15.150 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:36:15.151 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:36:15.151 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:36:15.151 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:15.151 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:36:15.157 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:36:15.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:36:15.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:15.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:16.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:16.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:16.144 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":572,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:16.168 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":572,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:17.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:17.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:18.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:18.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:19.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:19.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:19.169 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:19.169 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:20.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:20.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:20.172 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:20.203 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:20.203 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:36:20.203 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:20.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:36:21.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:21.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:21.205 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":578,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:21.229 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:21.229 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:36:22.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:22.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:22.234 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":578,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:22.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:22.256 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:36:23.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:23.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:23.260 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":578,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:23.282 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:36:23.283 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:23.283 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:36:24.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:24.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:24.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:36:25.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:25.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:25.403 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:36:25.403 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:25.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:26.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:26.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:26.413 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:26.436 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:26.436 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:36:26.436 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:36:26.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:26.437 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:36:27.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:27.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:27.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":578,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:27.428 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:27.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:36:28.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:28.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:28.410 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":578,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:28.443 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:28.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:36:29.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:29.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:29.447 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":578,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:29.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:36:29.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:36:29.469 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:36:29.469 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:36:29.470 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:36:29.470 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:36:29.484 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:36:29.484 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:36:29.484 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:29.485 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:30.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:30.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:30.617 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:30.640 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:31.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:31.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:32.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:32.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:33.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:33.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:33.641 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:33.641 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:34.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:34.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:34.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:34.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:35.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:35.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:36.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:36.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:37.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:37.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:37.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:37.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:38.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:38.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:38.659 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:38.683 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":578,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:39.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:39.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:40.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:40.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:41.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:41.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:41.683 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:41.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:36:42.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:42.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:43.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:43.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:44.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:44.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:44.481 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":582,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:44.503 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":582,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:36:44.503 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:36:44.504 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:44.504 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:36:45.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:45.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:45.510 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":582,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:45.529 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:36:45.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:36:46.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:46.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:46.546 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":582,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:36:46.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:36:47.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:47.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:48.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:48.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:49.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:49.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:50.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:50.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:51.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:51.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:52.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:52.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:53.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:53.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:54.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:54.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:55.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:55.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:56.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:56.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:57.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:57.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:58.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:58.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:36:59.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:36:59.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:00.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:00.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:01.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:01.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:02.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:02.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:03.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:03.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:04.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:04.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:05.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:05.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:06.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:06.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:07.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:07.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:08.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:08.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:09.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:09.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:10.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:10.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:11.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:11.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:12.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:12.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:13.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:13.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:14.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:14.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:15.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:15.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:16.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:16.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:17.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:17.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:18.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:18.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:19.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:19.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:20.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:20.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:21.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:21.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:22.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:22.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:23.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:23.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:24.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:24.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:25.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:25.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:26.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:26.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:27.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:27.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:28.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:28.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:29.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:29.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:30.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:30.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:31.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:31.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:31.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:31.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:33.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:33.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:34.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:34.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:35.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:35.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:36.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:36.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:37.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:37.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:38.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:38.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:39.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:39.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:40.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:40.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:41.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:41.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:42.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:42.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:43.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:43.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:44.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:44.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:45.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:45.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:46.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:46.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:47.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:47.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:48.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:48.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:49.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:49.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:50.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:50.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:51.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:51.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:52.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:52.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:53.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:53.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:54.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:54.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:55.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:55.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:56.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:56.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:57.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:57.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:58.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:58.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:37:59.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:37:59.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:00.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:00.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:01.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:01.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:02.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:02.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:03.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:03.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:04.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:04.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:05.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:05.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:06.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:06.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:07.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:07.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:08.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:08.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:09.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:09.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:10.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:10.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:11.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:11.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:12.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:12.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:13.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:13.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:14.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:14.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:15.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:15.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:16.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:16.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:16.570 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:38:16.571 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:38:16.571 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:38:16.571 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:16.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:38:16.577 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:38:16.577 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:38:16.577 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:16.577 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:17.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:17.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:17.530 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":582,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:17.553 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":582,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:18.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:18.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:19.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:19.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:20.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:20.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:20.554 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:20.555 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:21.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:21.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:22.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:22.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:23.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:23.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:24.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:24.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:25.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:25.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:25.454 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:25.485 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:25.485 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:38:25.485 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:25.485 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:38:26.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:26.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:26.462 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":588,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:26.488 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:26.488 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:38:27.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:27.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:27.710 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":588,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:27.744 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:27.744 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:38:28.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:28.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:28.759 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":588,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:28.782 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:38:28.782 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:28.782 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:38:29.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:29.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:29.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:38:30.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:30.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:30.869 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:38:30.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:30.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:31.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:31.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:31.903 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:31.925 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:31.925 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:38:31.925 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:38:31.926 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:31.926 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:38:32.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:32.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:32.994 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":588,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:33.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:33.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:38:33.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:33.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:33.994 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":588,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:34.027 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:34.027 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:38:34.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:34.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:35.023 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":588,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:35.044 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:38:35.044 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:38:35.044 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:38:35.045 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:38:35.045 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:38:35.045 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:38:35.063 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:38:35.063 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:38:35.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:35.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:35.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:35.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:36.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:36.148 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:36.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:36.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:37.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:37.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:38.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:38.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:39.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:39.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:39.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:39.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:40.157 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:40.189 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:40.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:40.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:41.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:41.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:42.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:42.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:43.190 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:43.190 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:43.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:43.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:44.178 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:44.210 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":588,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:44.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:44.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:45.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:45.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:46.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:46.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:47.211 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:47.212 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:38:47.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:47.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:48.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:48.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:49.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:49.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:50.388 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":592,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:50.419 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":592,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:38:50.419 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:38:50.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:50.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:38:50.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:50.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:51.394 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":592,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:51.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:51.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:51.418 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:38:51.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:38:52.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:52.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:52.434 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":592,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:38:52.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:38:53.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:53.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:54.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:54.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:55.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:55.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:56.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:56.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:57.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:57.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:58.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:58.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:38:59.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:38:59.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:00.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:00.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:01.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:01.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:02.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:02.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:03.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:03.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:04.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:04.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:05.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:05.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:06.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:06.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:07.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:07.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:08.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:08.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:09.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:09.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:10.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:10.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:11.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:11.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:12.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:12.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:13.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:13.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:14.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:14.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:15.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:15.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:16.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:16.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:17.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:17.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:18.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:18.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:19.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:19.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:20.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:20.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:21.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:21.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:22.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:22.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:23.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:23.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:24.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:24.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:25.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:25.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:26.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:26.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:27.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:27.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:28.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:28.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:29.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:29.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:30.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:30.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:31.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:31.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:32.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:32.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:33.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:33.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:34.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:34.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:35.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:35.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:36.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:36.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:37.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:37.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:38.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:38.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:39.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:39.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:40.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:40.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:41.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:41.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:42.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:42.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:43.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:43.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:44.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:44.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:45.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:45.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:46.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:46.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:47.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:47.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:48.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:48.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:49.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:49.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:50.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:50.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:51.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:51.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:52.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:52.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:53.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:53.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:54.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:54.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:55.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:55.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:56.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:56.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:57.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:57.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:58.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:58.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:39:59.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:39:59.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:00.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:00.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:01.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:01.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:02.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:02.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:03.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:03.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:04.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:04.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:05.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:05.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:06.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:06.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:07.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:07.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:08.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:08.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:09.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:09.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:10.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:10.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:11.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:11.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:12.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:12.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:13.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:13.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:14.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:14.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:15.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:15.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:16.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:16.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:17.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:17.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:18.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:18.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:19.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:19.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:20.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:20.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:21.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:21.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:22.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:22.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:22.457 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:40:22.458 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:40:22.458 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:40:22.458 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:22.458 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:40:22.466 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:40:22.466 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:40:22.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:22.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:23.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:23.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:23.570 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:23.593 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:23.593 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:40:23.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:23.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:40:24.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:24.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:24.631 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":594,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:24.663 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:24.663 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:40:25.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:25.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:25.692 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":594,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:25.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:25.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:40:26.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:26.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:26.756 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":594,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:26.784 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:40:26.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:26.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:40:27.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:27.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:27.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:40:28.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:28.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:28.980 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:40:28.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:28.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:29.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:29.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:29.986 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:30.021 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:30.021 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:40:30.021 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:40:30.022 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:30.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:40:30.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:30.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 30 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:30.985 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":594,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:31.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:31.017 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:40:31.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:31.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:32.012 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":594,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:32.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:32.045 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:40:32.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:32.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:33.051 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":594,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:33.073 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:40:33.073 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:40:33.074 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:40:33.074 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:40:33.074 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:40:33.074 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:40:33.086 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:40:33.086 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:40:33.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:33.086 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:33.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:33.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:34.162 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:34.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:34.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:34.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:35.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:35.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:36.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:36.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:37.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:37.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:37.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:37.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:38.204 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:38.227 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:38.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:38.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:39.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:39.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:40.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:40.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:41.228 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:41.228 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:41.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:41.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:42.238 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:42.269 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":594,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:42.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:42.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:43.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:43.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:44.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:44.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:45.271 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:45.271 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:40:45.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:45.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:46.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:46.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:47.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:47.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:48.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:48.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:48.365 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:48.394 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:40:48.394 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:40:48.395 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:48.395 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:40:49.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:49.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:49.407 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":598,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:49.439 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:40:49.440 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:40:50.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:50.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:50.408 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":598,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:40:50.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:40:51.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:51.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:52.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:52.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:53.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:53.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:54.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:54.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:55.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:55.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:56.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:56.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:57.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:57.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:58.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:58.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:40:59.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:40:59.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:00.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:00.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:01.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:01.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:02.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:02.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:03.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:03.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:04.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:04.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:05.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:05.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:06.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:06.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:07.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:07.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:08.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:08.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:09.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:09.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:10.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:10.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:11.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:11.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:12.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:12.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:13.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:13.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:14.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:14.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:15.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:15.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:16.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:16.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:17.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:17.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:18.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:18.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:19.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:19.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:20.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:20.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:21.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:21.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:22.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:22.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:23.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:23.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:24.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:24.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:25.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:25.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:26.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:26.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:27.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:27.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:28.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:28.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:29.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:29.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:30.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:30.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:31.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:31.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:32.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:32.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:33.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:33.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:33.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:33.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:34.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:34.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:35.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:35.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:36.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:36.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:37.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:37.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:38.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:38.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:39.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:39.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:40.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:40.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:41.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:41.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:42.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:42.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:43.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:43.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:44.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:44.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:45.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:45.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:46.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:46.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:47.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:47.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:48.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:48.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:49.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:49.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:50.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:50.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:51.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:51.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:52.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:52.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:53.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:53.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:54.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:54.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:55.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:55.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:56.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:56.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:57.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:57.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:58.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:58.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:41:59.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:41:59.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:00.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:00.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:01.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:01.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:02.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:02.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:03.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:03.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:04.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:04.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:05.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:05.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:06.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:06.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:07.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:07.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:08.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:08.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:09.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:09.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:10.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:10.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:11.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:11.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:12.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:12.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:13.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:13.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:14.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:14.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:15.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:15.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:16.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:16.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:17.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:17.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:18.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:18.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:19.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:19.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:20.431 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:42:20.432 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:42:20.432 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:42:20.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:20.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:42:20.439 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:42:20.439 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:42:20.439 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:20.440 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:20.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:20.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:21.443 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:21.465 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:21.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:21.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:22.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:22.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:23.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:23.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:24.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:24.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:24.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:24.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:25.435 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:25.456 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:25.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:25.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:26.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:26.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:27.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:27.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:28.457 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:28.457 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:28.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:28.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:29.477 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:29.504 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":598,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:29.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:29.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:30.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:30.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:31.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:31.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:32.504 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:32.504 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:32.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:32.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:33.493 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:33.523 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:33.523 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:42:33.523 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:33.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:42:33.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:33.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:34.521 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":600,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:34.542 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:34.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:42:34.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:34.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:35.533 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":600,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:35.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:35.562 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:42:35.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:35.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:36.554 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":600,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:36.572 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:42:36.572 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:36.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:42:36.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:36.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:37.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:42:37.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:37.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:38.722 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:42:38.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:38.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:38.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:38.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:39.694 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:39.724 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:39.725 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:42:39.725 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:42:39.725 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:39.725 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:42:39.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:39.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:40.727 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":600,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:40.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:40.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:42:40.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:40.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:41.754 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":600,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:41.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:41.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:42:41.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:41.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:42.750 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":600,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:42.781 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T04:42:42.781 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:42:42.781 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T04:42:42.781 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:42:42.781 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T04:42:42.782 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:42:42.796 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T04:42:42.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:42:42.796 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:42.797 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:42.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:42.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:43.861 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:43.883 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:43.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:43.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:45.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:45.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:45.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:45.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:46.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:46.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:47.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:47.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:47.883 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:47.914 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":600,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:48.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:48.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:49.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:49.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:50.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:50.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:50.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:50.914 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:42:51.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:51.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:52.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:52.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:53.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:53.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:54.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:54.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:55.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:55.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:56.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:56.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:56.864 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:56.896 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:42:56.897 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:42:56.897 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:56.897 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:42:57.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:57.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:57.911 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":604,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:57.941 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:42:57.941 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:42:58.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:58.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:42:58.945 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":604,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:42:58.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:42:59.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:42:59.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:00.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:00.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:01.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:01.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:02.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:02.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:03.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:03.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:04.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:04.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:04.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:04.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:06.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:05.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:06.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:06.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:07.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:07.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:08.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:08.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:09.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:09.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:10.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:10.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:12.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:11.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:13.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:13.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:14.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:14.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:15.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:15.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:16.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:16.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:17.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:17.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:18.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:18.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:19.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:19.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:20.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:20.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:21.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:21.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:22.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:22.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:23.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:23.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:24.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:24.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:25.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:25.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:26.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:26.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:27.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:27.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:28.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:28.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:29.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:29.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:30.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:30.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:31.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:31.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:32.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:32.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:32.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:32.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:33.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:33.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:35.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:35.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:36.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:36.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:37.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:37.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:38.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:38.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:39.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:39.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:40.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:40.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:41.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:41.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:42.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:42.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:43.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:43.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:44.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:44.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:45.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:45.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:46.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:46.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:47.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:47.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:47.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:47.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:49.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:48.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:50.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:50.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:51.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:51.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:52.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:52.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:53.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:53.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:54.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:54.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:55.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:55.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:56.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:56.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:57.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:57.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:58.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:58.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:43:59.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:43:59.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:00.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:00.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:01.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:01.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:02.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:02.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:03.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:03.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:04.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:04.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:05.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:05.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:06.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:06.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:07.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:07.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:08.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:08.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:09.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:09.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:10.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:10.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:11.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:11.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:12.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:12.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:13.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:13.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:14.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:14.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:15.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:15.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:16.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:16.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:17.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:17.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:18.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:18.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:19.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:19.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:20.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:20.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:21.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:21.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:22.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:22.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:23.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:23.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:24.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:24.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:25.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:25.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:26.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:26.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:27.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:27.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:28.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:28.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:28.968 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T04:44:28.969 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T04:44:28.969 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T04:44:28.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:28.969 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T04:44:28.975 INFO:tasks.ceph.mon.c:Started 2019-10-01T04:44:28.975 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:44:28.975 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:28.975 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:29.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:29.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:29.973 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:29.996 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:30.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:30.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:31.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:31.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:32.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:32.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:32.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:32.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:33.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:33.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:33.996 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:34.021 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:34.021 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:44:34.021 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:34.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:44:34.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:34.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:35.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":606,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:35.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:35.053 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:44:35.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:35.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:36.087 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":606,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:36.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:36.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:44:36.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:36.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:37.114 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":606,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:37.135 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:44:37.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:37.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:44:37.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:37.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:38.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:44:38.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:38.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:39.212 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:44:39.212 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:39.212 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:39.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:39.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:40.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:40.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:40.208 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:40.238 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:40.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:44:40.239 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:44:40.239 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:40.239 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:44:41.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:41.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:41.325 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":606,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:41.347 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:41.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:44:42.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:42.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:42.349 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":606,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:42.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:42.379 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:44:43.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:43.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:43.407 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":606,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:43.439 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:44:43.439 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:44:43.439 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:44:43.439 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:44:43.439 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:44:43.440 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:44:43.458 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:44:43.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:44:43.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:43.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:44.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:44.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:44.512 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:44.533 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:45.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:45.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:46.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:46.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:47.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:47.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:47.534 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:47.534 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:48.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:48.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:48.519 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:48.541 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":606,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:49.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:49.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:50.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:50.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:51.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:51.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:51.542 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:51.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:44:52.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:52.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:53.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:53.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:54.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:54.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:55.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:55.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:56.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:56.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:57.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:57.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:58.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:58.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:58.650 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:58.680 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:44:58.681 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:44:58.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:58.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:44:59.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:44:59.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:44:59.667 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":610,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:44:59.689 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:44:59.690 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:45:00.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:00.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:00.698 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":610,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:45:00.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:45:01.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:01.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:02.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:02.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:03.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:03.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:04.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:04.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:05.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:05.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:06.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:06.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:07.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:07.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:08.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:08.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:09.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:09.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:10.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:10.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:11.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:11.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:12.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:12.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:13.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:13.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:14.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:14.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:15.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:15.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:16.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:16.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:17.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:17.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:18.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:18.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:19.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:19.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:20.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:20.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:21.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:21.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:22.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:22.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:23.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:23.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:24.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:24.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:25.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:25.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:26.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:26.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:27.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:27.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:28.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:28.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:29.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:29.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:30.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:30.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:31.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:31.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:32.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:32.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:33.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:33.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:34.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:34.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:35.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:35.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:36.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:36.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:37.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:37.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:38.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:38.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:39.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:39.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:40.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:40.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:41.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:41.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:42.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:42.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:43.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:43.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:44.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:44.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:45.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:45.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:46.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:46.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:47.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:47.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:48.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:48.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:49.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:49.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:50.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:50.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:51.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:51.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:52.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:52.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:53.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:53.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:54.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:54.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:55.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:55.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:56.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:56.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:57.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:57.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:58.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:58.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:45:59.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:45:59.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:00.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:00.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:01.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:01.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:02.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:02.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:03.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:03.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:04.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:04.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:05.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:05.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:06.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:06.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:07.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:07.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:08.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:08.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:09.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:09.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:10.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:10.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:11.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:11.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:12.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:12.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:13.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:13.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:14.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:14.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:15.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:15.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:16.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:16.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:17.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:17.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:18.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:18.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:19.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:19.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:20.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:20.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:21.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:21.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:22.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:22.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:23.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:23.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:24.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:24.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:25.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:25.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:26.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:26.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:27.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:27.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:28.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:28.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:29.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:29.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:30.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:30.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:30.720 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:46:30.720 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:46:30.720 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:46:30.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:30.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:46:30.726 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:46:30.726 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:46:30.727 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:30.727 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:31.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:31.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:31.691 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:31.722 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:32.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:32.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:33.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:33.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:34.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:34.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:34.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:34.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:35.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:35.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:35.732 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:35.762 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:36.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:36.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:37.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:37.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:38.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:38.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:38.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:38.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:39.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:39.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:39.781 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:39.808 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:40.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:40.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:41.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:41.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:42.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:42.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:42.809 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:42.809 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:43.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:43.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:43.821 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:43.843 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:43.843 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:46:43.844 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:43.844 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:46:44.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:44.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:44.814 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":612,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:44.844 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:44.844 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:46:45.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:45.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:45.913 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":612,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:45.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:45.939 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:46:46.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:46.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:46.928 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":612,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:46.953 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:46:46.953 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:46.953 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:46:47.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:47.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:48.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:46:48.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:48.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:49.011 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:46:49.011 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:49.011 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:49.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:49.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:50.012 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:50.034 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:50.034 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:46:50.034 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:46:50.034 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:50.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:46:50.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:50.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:51.049 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":612,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:51.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:51.072 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:46:51.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:51.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:52.079 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":612,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:52.111 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:52.111 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:46:52.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:52.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:53.110 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":612,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:46:53.132 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:46:53.132 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:46:53.132 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:46:53.132 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:46:53.133 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:46:53.133 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:46:53.166 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:46:53.166 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:46:53.167 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:53.167 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:53.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:53.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:54.164 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:54.187 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:54.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:54.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:55.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:55.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:56.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:56.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:57.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:46:57.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:46:57.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:57.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:58.193 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:58.212 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:46:58.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:58.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:46:59.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:46:59.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:00.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:00.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:01.213 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:47:01.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:47:01.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:01.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:02.218 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:47:02.250 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":612,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:47:02.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:02.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:03.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:03.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:04.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:04.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:05.251 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:47:05.251 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:47:05.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:05.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:06.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:06.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:07.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:07.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:08.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:08.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:09.299 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:47:09.322 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:47:09.323 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:47:09.323 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:47:09.323 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:47:09.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:09.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:10.346 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":614,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:47:10.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:47:10.378 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:47:10.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:10.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:11.396 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":614,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:47:11.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:47:11.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:11.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:12.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:12.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:13.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:13.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:14.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:14.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:15.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:15.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:16.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:16.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:17.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:17.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:19.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:19.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:19.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:19.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:20.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:20.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:21.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:21.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:22.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:22.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:23.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:23.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:24.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:24.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:25.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:25.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:26.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:26.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:27.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:27.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:28.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:28.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:29.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:29.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:30.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:30.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:31.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:31.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:32.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:32.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:33.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:33.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:34.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:34.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:35.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:35.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:36.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:36.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:37.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:37.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:38.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:38.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:39.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:39.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:40.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:40.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:41.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:41.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:42.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:42.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:43.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:43.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:44.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:44.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:45.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:45.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:46.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:46.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:47.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:47.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:48.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:48.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:49.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:49.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:50.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:50.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:51.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:51.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:52.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:52.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:53.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:53.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:54.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:54.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:55.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:55.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:56.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:56.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:57.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:57.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:58.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:58.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:47:59.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:47:59.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:00.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:00.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:01.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:01.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:02.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:02.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:03.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:03.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:04.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:04.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:05.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:05.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:06.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:06.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:07.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:07.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:08.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:08.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:09.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:09.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:10.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:10.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:11.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:11.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:12.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:12.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:13.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:13.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:14.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:14.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:15.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:15.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:16.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:16.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:17.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:17.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:18.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:18.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:19.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:19.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:20.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:20.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:21.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:21.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:22.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:22.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:23.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:23.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:24.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:24.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:25.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:25.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:26.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:26.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:27.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:27.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:28.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:28.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:29.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:29.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:30.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:30.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:31.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:31.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:32.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:32.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:33.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:33.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:34.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:34.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:35.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:35.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:36.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:36.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:37.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:37.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:38.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:38.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:39.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:39.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:40.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:40.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:41.420 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:48:41.420 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:48:41.420 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:48:41.421 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:48:41.421 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:48:41.427 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:48:41.428 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:48:41.428 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:41.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:48:41.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:41.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:42.436 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:42.458 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:42.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:42.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:43.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:43.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:44.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:44.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:45.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:45.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:48:45.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:45.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:46.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:46.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:46.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:46.556 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:47.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:47.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:48.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:48.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:49.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:49.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:49.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:49.558 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:48:50.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:50.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:50.567 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:50.589 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:51.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:51.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:52.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:52.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:53.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:53.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:53.590 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:53.590 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:48:54.611 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:54.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:54.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:54.635 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":614,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:55.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:55.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:56.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:56.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:57.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:57.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:57.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:57.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:48:58.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:58.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:58.651 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:58.676 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:48:58.676 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:48:58.677 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:58.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:48:59.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:48:59.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:48:59.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":618,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:48:59.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:48:59.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:49:00.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:00.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:00.701 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":618,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:00.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:00.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:49:01.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:01.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:01.713 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":618,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:01.735 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:49:01.736 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:01.736 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:49:02.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:02.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:02.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:49:03.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:03.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:03.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:49:03.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:03.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:49:04.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:04.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:04.817 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:04.839 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:04.839 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:49:04.839 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:49:04.840 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:04.840 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:49:05.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:05.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:05.830 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":618,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:49:05.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:05.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:49:06.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:06.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:06.852 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":618,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:06.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:06.884 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:49:07.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:07.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:07.907 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":618,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:07.929 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:49:07.930 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:49:07.930 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:49:07.930 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:49:07.930 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:49:07.930 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:49:07.946 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:49:07.946 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:49:07.947 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:07.947 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:49:08.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:08.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:08.995 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:09.017 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:09.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:09.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:10.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:10.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:11.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:11.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:12.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:12.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:49:12.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:12.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:13.015 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:13.038 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":618,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:13.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:13.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:14.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:14.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:15.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:15.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:16.039 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:16.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:49:16.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:16.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:17.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:17.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:18.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:18.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:19.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:19.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:20.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:20.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:21.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:21.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:22.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:22.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:23.451 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:23.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:23.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:23.476 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:49:23.476 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:49:23.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:23.477 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:49:24.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:24.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:24.496 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":622,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:24.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:49:24.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:49:25.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:25.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:25.502 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":622,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:49:25.525 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:49:26.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:26.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:27.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:27.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:28.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:28.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:29.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:29.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:30.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:30.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:31.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:31.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:32.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:32.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:33.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:33.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:34.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:34.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:35.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:35.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:36.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:36.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:37.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:37.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:38.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:38.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:39.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:39.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:40.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:40.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:41.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:41.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:42.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:42.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:43.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:43.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:44.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:44.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:45.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:45.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:46.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:46.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:47.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:47.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:48.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:48.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:49.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:49.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:50.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:50.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:51.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:51.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:52.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:52.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:53.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:53.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:54.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:54.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:55.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:55.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:56.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:56.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:57.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:57.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:58.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:58.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:49:59.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:49:59.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:00.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:00.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:01.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:01.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:02.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:02.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:03.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:03.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:04.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:04.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:05.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:05.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:06.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:06.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:07.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:07.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:08.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:08.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:09.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:09.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:10.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:10.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:11.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:11.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:12.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:12.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:13.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:13.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:14.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:14.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:15.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:15.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:16.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:16.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:17.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:17.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:18.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:18.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:19.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:19.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:20.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:20.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:21.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:21.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:22.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:22.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:23.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:23.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:24.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:24.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:25.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:25.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:26.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:26.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:27.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:27.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:28.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:28.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:29.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:29.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:30.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:30.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:31.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:31.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:32.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:32.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:33.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:33.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:34.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:34.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:35.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:35.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:36.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:36.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:37.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:37.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:38.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:38.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:39.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:39.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:40.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:40.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:41.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:41.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:42.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:42.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:43.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:43.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:44.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:44.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:45.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:45.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:46.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:46.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:47.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:47.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:48.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:48.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:49.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:49.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:50.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:50.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:51.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:51.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:52.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:52.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:53.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:53.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:54.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:54.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:55.525 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:50:55.525 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:50:55.526 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:50:55.526 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:50:55.526 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:50:55.554 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:50:55.554 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:50:55.555 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:50:55.555 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:50:55.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:55.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:56.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:56.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:56.575 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:50:56.598 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:50:57.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:57.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:58.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:58.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:50:59.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:50:59.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:50:59.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:50:59.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:00.595 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:00.617 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:00.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:00.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:01.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:01.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:02.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:02.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:03.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:03.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:03.619 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:03.619 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:51:04.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:04.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:04.683 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:04.708 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":622,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:05.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:05.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:06.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:06.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:07.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:07.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:07.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:07.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:51:08.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:08.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:08.739 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:08.762 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:08.762 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:51:08.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:08.763 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:51:09.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:09.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:09.753 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":624,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:09.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:09.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:51:10.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:10.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:10.778 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":624,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:10.808 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:10.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:51:11.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:11.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:11.813 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":624,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:11.836 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:51:11.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:11.837 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:51:12.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:12.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:12.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:51:13.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:13.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:13.912 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:51:13.913 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:13.913 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:51:14.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:14.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:14.945 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:14.977 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:14.977 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:51:14.978 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:51:14.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:14.978 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:51:15.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:15.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:15.973 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":624,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:15.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:15.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:51:16.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:16.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:16.971 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":624,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:16.993 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:16.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:51:17.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:17.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:18.009 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":624,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:18.032 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:51:18.033 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:51:18.033 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:51:18.033 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:51:18.033 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:51:18.033 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:51:18.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:18.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:19.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:19.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:20.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:20.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:21.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:21.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:22.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:22.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:23.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:23.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:24.135 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:51:24.136 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:51:24.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:24.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:51:24.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:24.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:25.145 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:25.175 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:25.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:25.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:26.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:26.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:27.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:27.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:28.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:28.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:51:28.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:28.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:29.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:29.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:30.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:30.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:31.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:31.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:32.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:32.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:33.515 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:33.544 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:51:33.545 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:51:33.545 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:33.545 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:51:33.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:33.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:34.545 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":628,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:34.567 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:51:34.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:51:34.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:34.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:35.580 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":628,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:51:35.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:51:35.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:35.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:36.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:36.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:37.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:37.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:38.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:38.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:39.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:39.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:40.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:40.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:41.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:41.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:42.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:42.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:43.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:43.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:44.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:44.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:45.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:45.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:46.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:46.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:47.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:47.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:48.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:48.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:49.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:49.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:50.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:50.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:51.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:51.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:52.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:52.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:53.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:53.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:54.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:54.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:55.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:55.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:56.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:56.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:57.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:57.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:58.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:58.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:51:59.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:51:59.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:00.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:00.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:01.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:01.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:02.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:02.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:03.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:03.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:04.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:04.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:05.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:05.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:06.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:06.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:07.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:07.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:08.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:08.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:09.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:09.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:10.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:10.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:11.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:11.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:12.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:12.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:13.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:13.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:14.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:14.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:15.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:15.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:16.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:16.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:17.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:17.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:18.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:18.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:19.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:19.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:20.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:20.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:21.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:21.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:22.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:22.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:23.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:23.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:24.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:24.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:25.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:25.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:27.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:26.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:28.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:28.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:29.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:28.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:30.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:30.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:31.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:31.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:32.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:32.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:33.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:33.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:34.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:34.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:34.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:34.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:36.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:36.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:37.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:37.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:38.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:38.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:39.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:38.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:39.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:39.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:40.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:40.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:42.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:42.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:43.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:42.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:43.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:43.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:44.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:44.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:45.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:45.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:46.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:46.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:47.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:47.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:48.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:48.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:49.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:49.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:50.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:50.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:51.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:51.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:52.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:52.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:53.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:53.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:54.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:54.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:55.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:55.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:56.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:56.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:57.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:57.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:58.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:58.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:52:59.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:52:59.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:00.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:00.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:01.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:01.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:02.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:02.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:03.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:03.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:04.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:04.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:05.603 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:53:05.603 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:53:05.603 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:53:05.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:05.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:53:05.611 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:53:05.611 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:53:05.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:05.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:05.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:05.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:06.612 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:06.634 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:07.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:06.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:08.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:08.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:09.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:09.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:09.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:09.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:09.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:09.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:10.651 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:10.674 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:10.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:10.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:11.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:11.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:12.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:12.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:13.674 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:13.675 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:13.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:13.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:14.726 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:14.749 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":628,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:14.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:14.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:15.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:15.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:16.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:16.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:17.751 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:17.751 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:17.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:17.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:18.739 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:18.760 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:18.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:53:18.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:18.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:53:18.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:18.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:19.742 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":630,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:19.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:19.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:53:19.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:19.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:20.734 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":630,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:20.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:20.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:53:20.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:20.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:21.756 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":630,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:21.778 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:53:21.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:21.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:53:21.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:21.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:22.864 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:53:22.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:22.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:23.865 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:53:23.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:23.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:23.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:23.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:24.889 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:24.921 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:24.921 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:53:24.921 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:53:24.922 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:24.922 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:53:24.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:24.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:25.910 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":630,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:25.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:25.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:25.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:25.931 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:53:26.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:26.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:27.099 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":630,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:27.125 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:27.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:53:27.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:27.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:28.120 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":630,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:28.143 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:53:28.143 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:53:28.143 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:53:28.144 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:53:28.144 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:53:28.144 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:53:28.173 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:53:28.173 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:53:28.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:28.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:28.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:28.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:29.143 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:29.176 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:29.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:29.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:30.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:30.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:31.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:31.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:32.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:32.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:33.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:32.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:33.176 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:33.199 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:33.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:33.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:35.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:35.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:36.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:36.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:36.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:36.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:37.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:37.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:37.200 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:37.230 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":630,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:38.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:38.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:39.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:39.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:40.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:40.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:40.231 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:40.231 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:53:41.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:41.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:42.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:42.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:43.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:43.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:43.509 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:43.533 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:53:43.533 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:53:43.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:43.534 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:53:44.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:44.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:44.516 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":632,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:53:44.548 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:53:44.548 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:53:45.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:45.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:45.547 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":632,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:53:45.580 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:53:46.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:46.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:47.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:47.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:48.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:48.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:49.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:49.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:49.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:49.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:50.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:50.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:51.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:51.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:52.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:52.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:53.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:53.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:54.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:54.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:55.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:55.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:57.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:57.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:58.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:58.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:53:59.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:53:59.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:00.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:00.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:01.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:01.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:02.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:02.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:03.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:03.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:04.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:04.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:05.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:05.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:06.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:06.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:07.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:07.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:07.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:07.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:08.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:08.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:09.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:09.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:10.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:10.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:11.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:11.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:12.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:12.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:13.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:13.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:14.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:14.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:15.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:15.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:17.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:16.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:18.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:17.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:18.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:18.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:19.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:19.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:20.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:20.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:22.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:22.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:23.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:23.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:24.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:24.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:25.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:25.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:25.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:25.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:27.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:27.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:28.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:28.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:29.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:29.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:30.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:30.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:31.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:31.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:32.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:32.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:33.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:33.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:33.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:33.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:34.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:34.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:35.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:35.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:36.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:36.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:37.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:37.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:38.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:38.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:39.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:39.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:40.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:40.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:41.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:41.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:42.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:42.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:43.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:43.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:44.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:44.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:45.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:45.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:46.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:46.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:47.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:47.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:48.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:48.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:49.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:49.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:50.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:50.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:51.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:51.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:52.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:52.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:53.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:53.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:54.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:54.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:55.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:55.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:56.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:56.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:57.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:57.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:58.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:58.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:54:59.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:54:59.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:00.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:00.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:01.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:01.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:02.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:02.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:03.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:03.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:04.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:04.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:05.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:05.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:06.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:06.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:07.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:07.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:08.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:08.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:09.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:09.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:10.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:10.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:11.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:11.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:12.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:12.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:13.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:13.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:14.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:14.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:15.582 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:55:15.582 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:55:15.582 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:55:15.582 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:55:15.582 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:55:15.588 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:55:15.588 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:55:15.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:15.589 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:15.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:15.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:16.587 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:16.617 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:16.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:16.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:17.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:17.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:18.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:18.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:19.618 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:19.618 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:19.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:19.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:20.623 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:20.646 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:20.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:20.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:21.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:21.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:22.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:22.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:23.647 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:23.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:23.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:23.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:24.670 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:24.701 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:24.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:24.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:25.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:25.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:26.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:26.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:27.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:27.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:28.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:27.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:28.712 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:28.744 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":632,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:29.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:29.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:29.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:29.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 35 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:30.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:30.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:31.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:31.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:31.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:31.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:32.730 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:32.763 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:32.763 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:55:32.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:32.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:55:32.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:32.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:33.796 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":636,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:55:33.819 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:33.819 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:55:33.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:33.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:34.790 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":636,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:55:34.820 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:34.820 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:55:34.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:34.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:35.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":636,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:55:35.807 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:55:35.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:35.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:55:35.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:35.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:36.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:55:36.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:36.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:37.913 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:55:37.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:37.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:37.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:37.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:38.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:38.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:38.919 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:38.941 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:38.941 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:55:38.941 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:55:38.941 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:38.941 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:55:39.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:39.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:40.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":636,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:55:40.033 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:40.033 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:55:40.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:40.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:41.019 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":636,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:55:41.041 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:41.042 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:55:41.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:41.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:42.041 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":636,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:55:42.071 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T04:55:42.071 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T04:55:42.071 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T04:55:42.071 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:55:42.072 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T04:55:42.072 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:55:42.096 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T04:55:42.096 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:55:42.096 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:42.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:42.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:42.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:43.171 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:43.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:43.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:43.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:45.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:45.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:45.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:45.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:46.194 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:46.194 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:46.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:46.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:47.210 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:47.233 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:47.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:47.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:48.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:48.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:49.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:49.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:50.233 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:50.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:50.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:50.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:51.234 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:51.272 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":636,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:51.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:51.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:52.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:52.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:53.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:53.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:54.273 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:54.273 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:55:54.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:54.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:55.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:55.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:56.923 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":640,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:56.946 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":640,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:55:56.946 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:55:56.946 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:56.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:55:56.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:56.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:57.925 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":640,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:55:57.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:57.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:57.948 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:55:57.948 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:55:58.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:58.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:55:58.995 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":640,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:55:59.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:55:59.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:55:59.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:00.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:00.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:01.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:01.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:02.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:02.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:03.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:03.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:04.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:04.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:05.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:05.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:06.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:06.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:08.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:08.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:09.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:09.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:10.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:10.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:10.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:10.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:11.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:11.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:12.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:12.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:13.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:13.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:14.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:14.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:15.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:15.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:16.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:16.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:17.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:17.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:18.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:18.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:19.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:19.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:20.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:20.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:21.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:21.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:22.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:22.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:23.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:23.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:24.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:24.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:25.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:25.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:26.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:26.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:27.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:27.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:28.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:28.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:29.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:29.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:30.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:30.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:31.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:31.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:32.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:32.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:33.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:33.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:34.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:34.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:35.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:35.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:36.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:36.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:37.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:37.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:39.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:39.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:40.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:40.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:41.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:41.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:42.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:42.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:43.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:43.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:44.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:44.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:45.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:45.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:46.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:46.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:47.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:47.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:48.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:48.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:49.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:49.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:49.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:49.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:51.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:51.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:51.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:51.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:52.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:52.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:53.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:53.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:54.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:54.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:55.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:55.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:56.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:56.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:57.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:57.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:58.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:58.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:56:59.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:56:59.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:00.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:00.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:01.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:01.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:02.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:02.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:03.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:03.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:04.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:04.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:05.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:05.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:06.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:06.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:07.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:07.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:08.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:08.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:09.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:09.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:10.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:10.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:11.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:11.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:12.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:12.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:13.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:13.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:14.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:14.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:15.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:15.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:16.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:16.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:17.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:17.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:18.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:18.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:19.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:19.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:20.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:20.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:21.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:21.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:22.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:22.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:23.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:23.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:25.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:25.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:26.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:26.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:27.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:26.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:28.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:28.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:29.026 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T04:57:29.027 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T04:57:29.027 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T04:57:29.027 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:29.027 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T04:57:29.032 INFO:tasks.ceph.mon.a:Started 2019-10-01T04:57:29.033 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:57:29.033 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:29.033 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:57:29.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:29.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:30.007 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":640,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:30.030 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":640,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:30.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:30.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:31.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:31.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:32.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:32.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:33.031 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:33.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:57:33.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:33.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:34.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:34.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:35.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:35.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:36.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:36.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:37.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:37.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:38.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:38.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:38.565 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":648,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:38.585 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":648,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:38.585 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:57:38.586 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:38.586 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:57:39.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:39.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:39.480 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":648,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:39.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:39.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:57:40.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:40.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:40.494 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":648,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:40.526 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:40.526 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:57:41.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:41.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:41.543 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":648,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:41.565 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:57:41.565 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:41.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:57:42.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:42.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:42.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T04:57:43.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:43.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:43.682 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:57:43.683 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:43.683 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:57:44.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:44.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:44.669 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":648,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:44.698 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":648,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:57:44.698 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:57:44.698 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T04:57:44.699 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:44.699 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:57:45.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:45.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:45.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":648,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:45.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:45.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:57:46.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:46.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:46.634 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":648,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:46.656 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:57:46.656 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:57:47.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:47.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:47.652 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":648,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:57:47.673 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T04:57:47.673 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T04:57:47.673 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T04:57:47.673 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:57:47.674 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T04:57:47.674 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T04:57:47.714 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T04:57:47.715 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:57:47.715 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:57:47.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:57:48.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:48.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:49.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:49.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:50.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:50.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:51.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:51.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:52.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:52.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:53.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:53.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:54.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:54.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:55.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:55.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:56.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:56.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:57.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:57.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:58.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:58.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:57:59.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:57:59.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:00.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:00.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:01.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:01.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:02.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:02.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:02.716 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:58:02.716 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:58:02.717 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T04:58:02.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:58:02.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:58:03.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:03.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:04.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:04.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:05.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:05.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:06.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:06.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:07.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:07.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:08.375 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":650,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:58:08.406 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":650,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:58:08.407 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T04:58:08.407 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:58:08.407 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:58:08.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:08.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:09.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:09.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:09.457 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":650,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:58:09.479 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:58:09.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:58:10.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:10.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:10.494 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":650,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:58:10.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T04:58:11.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:11.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:12.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:12.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:13.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:13.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:14.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:14.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:15.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:15.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:16.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:16.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:17.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:17.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:18.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:18.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:19.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:19.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:20.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:20.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:21.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:21.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:22.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:22.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:23.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:23.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:24.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:24.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:25.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:25.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:26.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:26.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:27.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:27.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:28.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:28.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:29.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:29.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:30.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:30.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:31.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:31.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:32.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:32.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:33.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:33.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:34.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:34.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:35.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:35.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:36.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:36.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:37.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:37.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:38.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:38.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:39.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:39.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:40.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:40.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:41.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:41.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:42.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:42.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:43.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:43.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:44.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:44.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:45.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:45.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:46.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:46.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:47.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:47.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:47.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:47.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:49.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:49.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:49.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:49.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:50.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:50.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:51.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:51.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:52.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:52.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:53.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:53.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:54.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:54.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:55.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:55.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:56.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:56.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:57.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:57.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:58.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:58.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:58:59.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:58:59.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:00.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:00.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:01.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:01.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:02.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:02.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:03.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:03.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:04.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:04.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:05.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:05.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:06.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:06.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:07.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:07.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:08.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:08.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:09.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:09.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:10.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:10.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:11.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:11.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:12.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:12.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:13.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:13.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:14.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:14.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:15.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:15.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:16.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:16.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:17.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:17.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:18.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:18.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:19.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:19.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:20.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:20.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:21.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:21.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:22.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:22.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:23.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:23.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:24.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:24.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:25.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:25.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:26.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:26.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:27.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:27.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:28.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:28.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:29.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:29.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:30.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:30.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:31.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:31.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:32.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:32.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:33.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:33.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:34.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:34.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:35.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:35.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:36.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:36.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:37.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:37.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:38.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:38.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:39.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:39.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:40.528 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T04:59:40.528 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T04:59:40.528 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T04:59:40.528 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T04:59:40.528 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T04:59:40.534 INFO:tasks.ceph.mon.b:Started 2019-10-01T04:59:40.534 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T04:59:40.534 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T04:59:40.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T04:59:40.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:40.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:41.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:41.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:42.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:42.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:43.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:43.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:44.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:44.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:45.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:45.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:46.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:46.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:47.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:47.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:48.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:48.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:49.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:49.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:50.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:50.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:51.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:51.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:52.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:52.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:53.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:53.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:54.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:54.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:55.535 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T04:59:55.536 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T04:59:55.536 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T04:59:55.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:59:55.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T04:59:55.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:55.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:56.569 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":658,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:59:56.589 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":658,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T04:59:56.590 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T04:59:56.590 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:59:56.590 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T04:59:56.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:56.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:57.562 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":658,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:59:57.592 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:59:57.592 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T04:59:57.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:57.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:58.549 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":658,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T04:59:58.579 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:59:58.579 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T04:59:58.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:58.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T04:59:59.572 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":658,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T04:59:59.593 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T04:59:59.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T04:59:59.593 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T04:59:59.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T04:59:59.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:00.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:00:00.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:00.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:01.720 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:00:01.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:01.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:00:01.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:01.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:02.731 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":658,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:00:02.761 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":658,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:00:02.762 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:00:02.762 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:00:02.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:02.762 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:00:02.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:02.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:03.852 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":658,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:00:03.877 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:03.877 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:00:03.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:03.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:04.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":658,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:00:04.903 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:04.903 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:00:04.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:04.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:05.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:05.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:05.991 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":658,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:00:06.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:00:06.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T05:00:06.009 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:00:06.010 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:00:06.010 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:00:06.010 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:00:06.106 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:00:06.106 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T05:00:06.106 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T05:00:06.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:00:06.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:06.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:07.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:07.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:08.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:08.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:09.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:09.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:10.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:10.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:11.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:11.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:12.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:12.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:13.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:13.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:15.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:14.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:15.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:15.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:16.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:16.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:17.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:17.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:19.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:19.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:20.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:20.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:21.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:21.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:21.107 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T05:00:21.108 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T05:00:21.108 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:00:21.108 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:21.108 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:00:22.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:22.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:23.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:23.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:24.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:24.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:25.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:25.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:26.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:26.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:26.749 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":660,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:00:26.772 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":660,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:00:26.772 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:00:26.772 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:26.772 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:00:27.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:27.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:27.753 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":660,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:00:27.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:00:27.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:00:28.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:28.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:28.747 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":660,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:00:28.774 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:00:29.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:29.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:30.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:30.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:31.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:31.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:32.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:32.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:33.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:33.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:34.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:34.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:35.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:35.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:36.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:36.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:37.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:37.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:38.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:38.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:39.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:39.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:40.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:40.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:41.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:41.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:42.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:42.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:43.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:43.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:44.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:44.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:45.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:45.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:46.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:46.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:47.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:47.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:47.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:47.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:49.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:49.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:50.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:50.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:50.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:50.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:52.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:52.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:53.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:53.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:54.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:54.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:55.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:55.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:56.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:56.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:57.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:57.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:58.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:58.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:00:59.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:00:59.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:00.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:00.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:01.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:01.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:02.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:02.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:02.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:02.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:04.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:04.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:04.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:04.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:06.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:06.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:07.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:07.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:08.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:08.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:09.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:09.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:10.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:10.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:11.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:11.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:12.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:12.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:12.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:12.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:13.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:13.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:15.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:15.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:16.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:16.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:17.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:17.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:18.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:18.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:19.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:18.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:20.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:20.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:21.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:21.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:22.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:22.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:23.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:23.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:24.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:24.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:25.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:25.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:26.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:26.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:26.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:26.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:28.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:28.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:29.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:29.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:30.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:30.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:31.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:31.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:32.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:32.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:33.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:33.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:34.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:34.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:35.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:35.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:36.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:36.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:37.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:37.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:38.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:38.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:39.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:39.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:40.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:40.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:41.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:41.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:42.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:42.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:43.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:43.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:44.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:44.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:45.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:45.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:46.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:46.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:47.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:47.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:48.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:48.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:49.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:49.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:50.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:50.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:51.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:51.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:52.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:52.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:53.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:53.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:54.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:54.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:55.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:55.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:56.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:56.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:57.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:57.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:58.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:58.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:01:58.775 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:01:58.775 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:01:58.775 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:01:58.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:01:58.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:01:58.783 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:01:58.783 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T05:01:58.783 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T05:01:58.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:01:59.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:01:59.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:00.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:00.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:01.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:01.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:02.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:02.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:03.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:03.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:04.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:04.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:05.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:05.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:06.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:06.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:07.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:07.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:08.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:08.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:09.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:09.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:10.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:10.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:11.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:11.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:12.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:12.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:13.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:13.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:13.784 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T05:02:13.784 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T05:02:13.785 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:02:13.785 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:13.785 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:02:14.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:14.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:14.773 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":664,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:14.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":664,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:14.805 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:02:14.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:14.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:02:15.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:15.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:15.882 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":664,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:15.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:15.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:02:16.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:16.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:16.878 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":664,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:16.899 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:16.899 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:02:17.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:17.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:17.871 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":664,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:17.901 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:02:17.901 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:17.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:02:18.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:18.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:19.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:02:19.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:19.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:20.054 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:02:20.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:20.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:02:20.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:20.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:20.965 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":664,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:20.997 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":664,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:20.997 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:02:20.997 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:02:20.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:20.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:02:21.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:21.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:21.974 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":664,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:22.003 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:22.003 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:02:22.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:22.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:22.977 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":664,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:22.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:22.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:02:23.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:23.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:23.951 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":664,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:23.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:02:23.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T05:02:23.979 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:02:23.979 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:02:23.979 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:02:23.979 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:02:23.997 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:02:23.997 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:02:23.998 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:02:23.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:02:24.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:24.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:25.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:25.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:26.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:26.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:27.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:27.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:28.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:28.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:29.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:29.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:30.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:30.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:31.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:31.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:32.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:32.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:33.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:33.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:34.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:34.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:35.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:35.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:36.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:36.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:37.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:37.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:38.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:38.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:38.998 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:02:38.999 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:02:38.999 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:02:38.999 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:38.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:02:39.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:39.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:40.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:40.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:41.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:41.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:42.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:42.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:43.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:43.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:44.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:44.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:45.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:45.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:46.212 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":666,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:46.234 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":666,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:02:46.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:02:46.235 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:46.235 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:02:46.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:46.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:47.253 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":666,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:47.275 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:02:47.275 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:02:47.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:47.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:48.288 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":666,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:02:48.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:48.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:48.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:02:49.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:49.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:50.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:50.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:51.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:51.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:52.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:52.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:53.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:53.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:54.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:54.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:55.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:55.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:56.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:56.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:57.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:57.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:58.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:58.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:02:59.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:02:59.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:00.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:00.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:01.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:01.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:02.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:02.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:03.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:03.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:04.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:04.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:05.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:05.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:06.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:06.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:07.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:07.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:08.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:08.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:09.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:09.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:10.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:10.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:11.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:11.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:12.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:12.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:13.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:13.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:14.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:14.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:15.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:15.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:16.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:16.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:17.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:17.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:18.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:18.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:19.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:19.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:20.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:20.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:21.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:21.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:22.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:22.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:23.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:23.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:24.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:24.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:25.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:25.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:26.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:26.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:27.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:27.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:28.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:28.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:29.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:29.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:30.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:30.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:31.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:31.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:32.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:32.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:33.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:33.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:34.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:34.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:35.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:35.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:36.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:36.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:37.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:37.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:38.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:38.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:39.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:39.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:40.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:40.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:41.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:41.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:42.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:42.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:43.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:43.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:44.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:44.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:45.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:45.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:46.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:46.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:47.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:47.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:48.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:48.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:49.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:49.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:50.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:50.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:51.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:51.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:52.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:52.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:53.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:53.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:54.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:54.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:55.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:55.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:56.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:56.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:57.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:57.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:58.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:58.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:03:59.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:03:59.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:00.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:00.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:01.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:01.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:02.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:02.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:03.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:03.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:04.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:04.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:05.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:05.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:06.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:06.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:07.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:07.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:08.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:08.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:09.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:09.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:10.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:10.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:11.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:11.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:12.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:12.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:13.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:13.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:14.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:14.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:15.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:15.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:16.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:16.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:17.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:17.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:18.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:18.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:18.312 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:04:18.312 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:04:18.312 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:04:18.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:18.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:04:18.318 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:04:18.319 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:04:18.319 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:04:18.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:04:19.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:19.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:20.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:20.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:21.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:21.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:22.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:22.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:23.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:23.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:24.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:24.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:25.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:25.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:26.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:26.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:27.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:27.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:28.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:28.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:29.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:29.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:30.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:30.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:31.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:31.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:32.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:32.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:33.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:33.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:33.320 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:04:33.320 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:04:33.321 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:04:33.321 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:33.321 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:34.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:34.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:34.341 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:34.362 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:34.362 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:04:34.362 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:34.363 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:04:35.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:35.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:35.363 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":670,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:35.393 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:35.393 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:04:36.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:36.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:36.367 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":670,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:36.397 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:36.398 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:04:37.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:37.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:37.382 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":670,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:37.408 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:04:37.408 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:37.408 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:04:38.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:38.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:38.516 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:04:39.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:39.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:39.517 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:04:39.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:39.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:40.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:40.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:40.532 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:40.554 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:40.554 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:04:40.554 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:04:40.554 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:40.555 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:04:41.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:41.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:41.581 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":670,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:41.613 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:41.613 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:04:42.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:42.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:42.603 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":670,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:42.634 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:42.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:04:43.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:43.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:43.646 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":670,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:43.668 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:04:43.668 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:04:43.668 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:04:43.668 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:04:43.668 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:04:43.668 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:04:43.692 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:04:43.692 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:04:43.692 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:43.692 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:44.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:44.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:44.678 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:44.701 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:45.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:45.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:46.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:46.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:47.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:47.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:47.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:47.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:48.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:48.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:48.690 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:48.722 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:49.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:49.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:50.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:50.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:51.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:51.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:51.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:51.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:52.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:52.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:52.719 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:52.748 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":670,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:53.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:53.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:54.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:54.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:55.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:55.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:55.749 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:55.749 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:04:56.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:56.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:57.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:57.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:58.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:58.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:58.927 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:58.957 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:04:58.957 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:04:58.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:58.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:04:59.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:04:59.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:04:59.958 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":672,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:04:59.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:04:59.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:05:00.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:00.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:00.949 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":672,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:05:00.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:05:01.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:01.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:02.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:02.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:03.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:03.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:04.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:04.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:05.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:05.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:06.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:06.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:07.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:07.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:08.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:08.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:09.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:09.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:10.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:10.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:11.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:11.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:12.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:12.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:13.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:13.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:14.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:14.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:15.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:15.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:16.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:16.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:17.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:17.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:18.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:18.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:19.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:19.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:20.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:20.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:21.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:21.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:22.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:22.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:23.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:23.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:24.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:24.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:25.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:25.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:26.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:26.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:27.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:27.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:28.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:28.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:29.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:29.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:30.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:30.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:31.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:31.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:32.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:32.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:33.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:33.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:34.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:34.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:35.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:35.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:36.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:36.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:37.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:37.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:38.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:38.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:39.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:39.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:40.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:40.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:41.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:41.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:42.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:42.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:43.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:43.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:44.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:44.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:45.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:45.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:46.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:46.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:47.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:47.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:48.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:48.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:49.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:49.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:50.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:50.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:51.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:51.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:52.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:52.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:53.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:53.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:54.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:54.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:55.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:55.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:56.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:56.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:57.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:57.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:58.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:58.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:05:59.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:05:59.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:00.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:00.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:01.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:01.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:02.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:02.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:03.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:03.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:04.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:04.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:05.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:05.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:06.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:06.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:07.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:07.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:08.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:08.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:09.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:09.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:10.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:10.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:11.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:11.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:12.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:12.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:13.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:13.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:14.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:14.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:15.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:15.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:16.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:16.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:17.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:17.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:18.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:18.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:19.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:19.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:20.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:20.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:21.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:21.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:22.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:22.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:23.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:23.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:24.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:24.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:25.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:25.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:26.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:26.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:27.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:27.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:28.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:28.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:29.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:29.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:30.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:30.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:30.971 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:06:30.971 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:06:30.971 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:06:30.972 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:06:30.972 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:06:30.978 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:06:30.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:06:30.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:30.978 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:31.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:31.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:31.981 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:32.010 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:32.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:32.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:33.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:33.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:34.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:34.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:35.011 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:35.011 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:35.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:35.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:35.980 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:36.004 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":672,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:36.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:36.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:37.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:37.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:38.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:38.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:39.005 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:39.005 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:39.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:39.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:40.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:40.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:41.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:41.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:42.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:42.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:43.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:43.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:43.905 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":676,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:43.927 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":676,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:44.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:44.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:45.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:45.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:46.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:46.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:46.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:46.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:47.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:47.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:47.925 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":676,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:47.954 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":676,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:48.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:48.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:49.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:49.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:50.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:50.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:50.955 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:50.955 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:51.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:51.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:51.964 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:51.996 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:51.996 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:06:51.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:51.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:06:52.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:52.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:52.991 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":680,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:06:53.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:53.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:06:53.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:53.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:54.013 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":680,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:06:54.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:54.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:06:54.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:54.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:55.029 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":680,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:06:55.052 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:06:55.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:55.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:06:55.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:55.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:56.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:06:56.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:56.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:57.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:06:57.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:57.164 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:06:57.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:57.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:58.164 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:58.196 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:06:58.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:06:58.196 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:06:58.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:58.197 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:06:58.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:58.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:06:59.188 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":680,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:06:59.210 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:06:59.210 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:06:59.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:06:59.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:00.246 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":680,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:07:00.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:00.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:00.273 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:00.273 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:07:01.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:01.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:01.264 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":680,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:07:01.287 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:07:01.287 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:07:01.287 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:07:01.287 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:07:01.287 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:07:01.288 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:07:01.309 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:07:01.309 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:07:01.310 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:01.310 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:07:02.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:02.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:02.353 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:02.376 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:03.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:03.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:04.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:04.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:05.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:05.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:05.377 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:05.377 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:07:06.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:06.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:06.368 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:06.388 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":680,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:07.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:07.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:08.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:08.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:09.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:09.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:09.389 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:09.389 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:07:10.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:10.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:11.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:11.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:12.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:12.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:13.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:13.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:14.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:14.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:15.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:15.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:15.366 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":684,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:15.398 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":684,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:07:15.399 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:07:15.399 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:15.399 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:07:16.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:16.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:16.390 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":684,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:07:16.417 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:07:16.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:07:17.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:17.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:17.419 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":684,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:07:17.440 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:07:17.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:17.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:18.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:18.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:19.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:19.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:20.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:20.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:21.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:21.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:22.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:22.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:23.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:23.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:24.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:24.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:25.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:25.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:26.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:26.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:27.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:27.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:28.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:28.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:29.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:29.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:30.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:30.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:31.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:31.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:32.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:32.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:33.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:33.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:34.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:34.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:35.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:35.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:36.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:36.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:37.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:37.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:38.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:38.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:39.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:39.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:40.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:40.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:41.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:41.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:42.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:42.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:43.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:43.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:44.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:44.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:45.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:45.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:46.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:46.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:47.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:47.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:48.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:48.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:49.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:49.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:50.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:50.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:51.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:51.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:52.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:52.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:53.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:53.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:54.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:54.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:55.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:55.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:56.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:56.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:57.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:57.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:58.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:58.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:07:59.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:07:59.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:00.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:00.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:01.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:01.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:02.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:02.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:03.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:03.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:04.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:04.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:05.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:05.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:06.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:06.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:07.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:07.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:08.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:08.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:09.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:09.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:10.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:10.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:11.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:11.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:12.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:12.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:13.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:13.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:14.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:14.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:15.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:15.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:16.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:16.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:17.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:17.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:18.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:18.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:19.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:19.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:20.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:20.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:21.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:21.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:22.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:22.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:23.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:23.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:24.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:24.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:25.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:25.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:26.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:26.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:27.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:27.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:28.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:28.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:29.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:29.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:30.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:30.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:31.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:31.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:32.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:32.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:33.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:33.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:34.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:34.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:35.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:35.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:36.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:36.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:37.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:37.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:38.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:38.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:39.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:39.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:40.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:40.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:41.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:41.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:42.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:42.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:43.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:43.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:44.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:44.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:45.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:45.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:46.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:46.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:47.441 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:08:47.441 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:08:47.441 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:08:47.441 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:08:47.441 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:08:47.447 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:08:47.447 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:08:47.447 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:08:47.447 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:08:47.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:47.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:48.403 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":684,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:08:48.426 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":684,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:08:48.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:48.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:49.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:49.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:50.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:50.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:51.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:08:51.427 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:08:51.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:51.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:52.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:52.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:53.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:53.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:54.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:54.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:55.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:55.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:56.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:56.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:57.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:57.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:59.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:59.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:08:59.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:08:59.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:00.347 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:00.377 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:00.378 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:09:00.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:00.378 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:09:01.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:00.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:01.362 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":696,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:01.391 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:01.391 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:09:01.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:01.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:02.343 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":696,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:02.366 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:02.366 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:09:02.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:02.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:03.352 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":696,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:03.377 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:09:03.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:03.378 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:09:04.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:04.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:04.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:09:05.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:05.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:05.501 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:09:05.501 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:05.501 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:09:06.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:06.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:06.554 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:06.584 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:06.584 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:09:06.584 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:09:06.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:06.585 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:09:07.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:07.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:07.641 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":696,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:07.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:07.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:09:08.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:08.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:08.671 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":696,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:08.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:08.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:09:09.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:09.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:09.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":696,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:09.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:09:09.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:09:09.708 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:09:09.708 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:09:09.709 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:09:09.709 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:09:09.723 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:09:09.724 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:09:09.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:09.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:09:10.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:10.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:10.792 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:10.818 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:11.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:11.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:12.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:12.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:13.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:13.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:13.819 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:13.819 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:09:14.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:14.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:14.811 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:14.835 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:15.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:15.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:16.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:16.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:17.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:17.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:17.836 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:17.836 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:09:18.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:18.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:18.840 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:18.863 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:19.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:19.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:19.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:19.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:20.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:20.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:21.864 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:21.865 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:09:21.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:21.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:22.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:22.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:24.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:24.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:24.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:24.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:25.909 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:25.939 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:09:25.939 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:09:25.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:25.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:09:25.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:25.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:26.942 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":700,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:26.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:09:26.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:09:27.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:27.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:27.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:27.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:28.042 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":700,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:09:28.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:09:28.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:28.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:29.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:29.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:30.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:30.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:31.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:31.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:32.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:32.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:33.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:33.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:34.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:34.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:36.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:36.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:36.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:36.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:37.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:37.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:38.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:38.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:40.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:39.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:41.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:41.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:42.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:42.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:43.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:43.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:44.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:44.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:45.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:45.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:46.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:46.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:47.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:47.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:48.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:48.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:49.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:49.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:50.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:50.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:51.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:51.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:52.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:52.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:53.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:53.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:54.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:54.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:55.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:55.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:56.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:56.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:57.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:57.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:57.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:57.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:59.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:59.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:09:59.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:09:59.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:00.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:00.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:01.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:01.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:02.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:02.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:03.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:03.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:04.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:04.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:05.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:05.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:06.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:06.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:07.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:07.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:08.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:08.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:09.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:09.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:10.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:10.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:11.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:11.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:12.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:12.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:14.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:13.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:14.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:14.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:15.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:15.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:17.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:17.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:17.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:17.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:19.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:19.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:20.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:19.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:21.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:21.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:22.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:22.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:22.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:22.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:24.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:24.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:24.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:24.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:25.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:25.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:26.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:26.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:27.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:27.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:28.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:28.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:30.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:30.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:30.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:30.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:32.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:32.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:33.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:33.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:34.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:34.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:35.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:35.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:36.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:36.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:37.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:37.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:38.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:38.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:39.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:39.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:40.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:40.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:41.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:41.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:42.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:42.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:43.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:43.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:44.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:44.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:45.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:45.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:46.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:46.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:46.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:46.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:48.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:48.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:49.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:49.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:50.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:50.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:50.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:50.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:51.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:51.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:53.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:53.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:54.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:54.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:55.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:55.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:56.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:56.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:57.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:57.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:58.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:58.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:58.066 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:10:58.066 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:10:58.066 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:10:58.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:10:58.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:10:58.074 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:10:58.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:10:58.074 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:10:58.074 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:10:59.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:10:59.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:10:59.056 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:10:59.086 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:00.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:00.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:01.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:01.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:02.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:02.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:02.087 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:02.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:03.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:03.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:03.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:03.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:05.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:05.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:06.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:06.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:07.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:07.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:07.650 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:07.673 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:07.673 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:11:07.673 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:07.674 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:11:08.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:08.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:08.675 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":708,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:08.707 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:08.707 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:11:09.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:09.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:09.709 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":708,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:09.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:09.730 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:11:10.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:10.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:10.724 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":708,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:10.753 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:11:10.753 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:10.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:11:10.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:10.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:11.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:11:12.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:11.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:12.840 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:11:12.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:12.841 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:12.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:12.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:13.848 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:13.867 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:13.868 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:11:13.868 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:11:13.868 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:13.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:11:13.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:13.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:14.861 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":708,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:14.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:14.892 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:11:15.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:15.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:15.882 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":708,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:15.913 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:15.914 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:11:16.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:16.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:16.953 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":708,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:16.971 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:11:16.972 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:11:16.972 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:11:16.972 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:11:16.972 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:11:16.972 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:11:17.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:17.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:17.018 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:11:17.018 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:11:17.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:17.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:18.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:18.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:18.091 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:18.113 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:19.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:19.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:20.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:20.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:21.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:21.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:21.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:21.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:21.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:21.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:22.112 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:22.135 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:22.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:22.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:23.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:23.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:24.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:24.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:25.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:25.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:25.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:25.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:26.160 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:26.192 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":708,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:26.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:26.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:27.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:27.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:29.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:29.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:29.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:29.192 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:11:30.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:30.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:30.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:30.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:32.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:31.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:32.621 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":712,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:32.643 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":712,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:11:32.644 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:11:32.644 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:32.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:11:32.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:32.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:33.627 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":712,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:33.660 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:11:33.661 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:11:33.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:33.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:34.665 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":712,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:11:34.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:11:34.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:34.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:35.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:35.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:36.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:36.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:38.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:38.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:39.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:39.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:40.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:40.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:41.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:41.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:42.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:42.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:43.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:43.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:44.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:44.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:45.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:45.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:45.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:45.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:46.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:46.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:47.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:47.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:48.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:48.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:49.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:49.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:50.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:50.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:51.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:51.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:52.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:52.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:53.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:53.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:54.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:54.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:55.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:55.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:56.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:56.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:57.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:57.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:58.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:58.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:11:59.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:11:59.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:00.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:00.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:01.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:01.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:02.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:02.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:03.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:03.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:04.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:04.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:05.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:05.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:06.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:06.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:08.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:08.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:09.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:09.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:10.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:10.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:11.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:11.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:12.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:12.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:13.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:13.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:14.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:14.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:15.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:15.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:16.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:16.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:17.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:17.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:18.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:18.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:19.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:19.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:20.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:20.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:21.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:21.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:22.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:22.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:23.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:23.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:24.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:24.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:25.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:25.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:26.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:26.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:27.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:27.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:28.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:28.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:29.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:29.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:30.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:30.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:31.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:31.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:32.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:32.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:33.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:33.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:34.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:34.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:35.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:35.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:36.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:36.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:37.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:37.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:38.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:38.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:39.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:39.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:40.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:40.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:41.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:41.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:42.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:42.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:43.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:43.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:44.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:44.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:45.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:45.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:46.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:46.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:47.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:47.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:48.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:48.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:49.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:49.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:50.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:50.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:51.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:51.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:51.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:51.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:52.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:52.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:53.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:53.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:54.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:54.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:55.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:55.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:56.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:56.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:57.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:57.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:58.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:58.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:12:59.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:12:59.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:00.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:00.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:01.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:01.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:02.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:02.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:04.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:04.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:04.688 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:13:04.688 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:13:04.688 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:13:04.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:04.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:13:04.694 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:13:04.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:13:04.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:04.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:13:04.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:04.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:05.670 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":712,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:05.692 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":712,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:05.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:05.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:07.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:07.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:08.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:08.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:08.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:08.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:13:08.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:08.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:10.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:10.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:10.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:10.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:11.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:11.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:12.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:12.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:13.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:13.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:15.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:14.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:15.942 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:15.965 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:15.965 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:13:15.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:15.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:13:16.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:16.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:17.002 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":718,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:17.026 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:17.026 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:13:17.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:17.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:17.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:17.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:18.024 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":718,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:18.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:18.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:13:18.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:18.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:19.048 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":718,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:19.069 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:13:19.069 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:19.069 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:13:19.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:19.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:20.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:13:20.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:20.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:21.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:13:21.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:21.201 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:13:21.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:21.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:22.209 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:22.241 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:22.241 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:13:22.241 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:13:22.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:22.242 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:13:22.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:22.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:23.230 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":718,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:23.262 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:23.263 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:13:23.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:23.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:24.266 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":718,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:24.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:24.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:13:24.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:24.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:25.286 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":718,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:25.309 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:13:25.309 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:13:25.309 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:13:25.309 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:13:25.310 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:13:25.310 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:13:25.329 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:13:25.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:13:25.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:25.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:13:25.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:25.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:26.406 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:26.428 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":718,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:26.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:26.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:27.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:27.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:29.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:29.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:29.430 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:29.430 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:13:30.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:30.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:31.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:31.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:32.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:32.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:33.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:33.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:34.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:34.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:35.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:35.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:36.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:36.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:37.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:37.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:38.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:38.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:39.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:39.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:40.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:40.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:40.130 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":722,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:40.153 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":722,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:13:40.153 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:13:40.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:40.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:13:41.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:41.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:41.161 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":722,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:41.191 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:13:41.191 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:13:42.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:42.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:42.156 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":722,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:13:42.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:13:43.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:43.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:44.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:44.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:45.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:45.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:46.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:46.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:47.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:47.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:48.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:48.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:49.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:49.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:50.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:50.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:51.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:51.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:52.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:52.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:53.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:53.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:54.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:54.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:55.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:55.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:56.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:56.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:57.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:57.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:58.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:58.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:13:59.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:13:59.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:00.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:00.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:01.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:01.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:02.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:02.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:03.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:03.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:04.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:04.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:05.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:05.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:06.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:06.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:07.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:07.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:08.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:08.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:09.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:09.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:10.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:10.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:11.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:11.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:12.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:12.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:13.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:13.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:14.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:14.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:15.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:15.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:16.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:16.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:17.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:17.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:18.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:18.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:19.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:19.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:20.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:20.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:21.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:21.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:22.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:22.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:23.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:23.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:24.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:24.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:25.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:25.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:26.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:26.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:27.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:27.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:28.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:28.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:29.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:29.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:30.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:30.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:31.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:31.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:32.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:32.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:33.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:33.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:34.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:34.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:35.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:35.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:36.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:36.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:37.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:37.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:38.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:38.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:39.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:39.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:40.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:40.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:41.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:41.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:42.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:42.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:43.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:43.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:44.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:44.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:45.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:45.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:46.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:46.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:47.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:47.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:48.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:48.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:49.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:49.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:50.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:50.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:51.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:51.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:52.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:52.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:53.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:53.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:54.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:54.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:55.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:55.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:56.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:56.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:57.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:57.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:58.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:58.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:14:59.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:14:59.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:00.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:00.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:01.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:01.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:02.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:02.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:03.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:03.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:04.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:04.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:05.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:05.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:06.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:06.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:07.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:07.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:08.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:08.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:09.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:09.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:09.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:09.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:10.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:10.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:11.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:11.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:12.179 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:15:12.179 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:15:12.179 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:15:12.180 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:12.180 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:15:12.186 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:15:12.186 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:15:12.186 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:12.187 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:12.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:12.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:13.139 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":722,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:13.170 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":722,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:13.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:13.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:14.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:14.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:15.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:15.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:16.172 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:16.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:16.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:16.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:17.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:17.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:18.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:18.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:19.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:19.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:20.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:20.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:21.135 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:21.165 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:21.165 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:15:21.166 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:21.166 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:15:21.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:21.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:22.185 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":728,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:22.203 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:22.203 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:15:23.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:22.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:23.158 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":728,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:23.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:23.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:15:23.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:23.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:24.159 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":728,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:24.177 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:15:24.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:24.178 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:15:24.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:24.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:25.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:15:26.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:26.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:26.302 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:15:26.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:26.302 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:27.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:27.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:27.291 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:27.323 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:27.323 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:15:27.323 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:15:27.323 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:27.323 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:15:28.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:28.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:28.330 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":728,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:28.361 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:28.362 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:15:29.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:29.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:29.350 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":728,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:29.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:29.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:15:30.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:30.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:30.372 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":728,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:30.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:15:30.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:15:30.404 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:15:30.405 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:15:30.405 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:15:30.405 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:15:30.424 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:15:30.424 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:15:30.425 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:30.425 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:31.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:31.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:31.472 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:31.490 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:32.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:32.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:33.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:33.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:34.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:34.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:34.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:34.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:35.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:35.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:35.501 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:35.532 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:36.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:36.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:36.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:36.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:37.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:37.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:38.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:38.534 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:38.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:38.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:39.524 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:39.542 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":728,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:39.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:39.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:40.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:40.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:41.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:41.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:42.543 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:42.543 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:15:42.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:42.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:43.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:43.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:44.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:44.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:45.381 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":732,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:45.404 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":732,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:15:45.404 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:15:45.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:45.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:15:45.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:45.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:46.399 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":732,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:46.430 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:15:46.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:15:47.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:46.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:47.399 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":732,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:15:47.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:15:47.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:47.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:48.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:48.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:49.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:49.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:50.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:50.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:51.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:51.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:53.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:53.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:54.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:54.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:55.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:55.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:56.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:56.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:57.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:57.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:58.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:58.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:15:59.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:15:59.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:00.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:00.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:01.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:01.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:02.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:02.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:03.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:03.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:04.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:03.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:05.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:05.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:06.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:06.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:07.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:07.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:08.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:08.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:09.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:09.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:10.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:10.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:11.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:11.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:12.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:12.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:13.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:13.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:14.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:14.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:15.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:15.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:15.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:15.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:17.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:16.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:17.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:17.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:18.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:18.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:19.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:19.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:20.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:20.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:21.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:21.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:23.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:23.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:24.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:24.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:25.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:25.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:26.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:26.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:27.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:27.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:28.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:28.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:29.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:29.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:30.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:30.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:31.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:30.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:32.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:32.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:33.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:33.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:34.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:34.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:35.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:35.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:36.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:36.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:37.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:37.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:38.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:38.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:39.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:39.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:40.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:40.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:41.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:41.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:42.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:42.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:43.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:43.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:44.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:44.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:45.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:45.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:46.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:46.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:47.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:47.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:48.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:48.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:49.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:49.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:50.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:50.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:51.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:51.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:52.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:52.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:53.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:53.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:54.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:54.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:55.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:55.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:56.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:56.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:57.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:57.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:58.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:58.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:16:59.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:16:59.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:00.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:00.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:01.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:01.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:02.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:02.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:03.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:03.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:04.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:04.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:05.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:05.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:06.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:06.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:07.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:07.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:08.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:08.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:09.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:09.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:10.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:10.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:11.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:11.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:12.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:12.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:13.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:13.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:14.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:14.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:15.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:15.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:16.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:16.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:17.422 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:17:17.423 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:17:17.423 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:17:17.423 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:17.423 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:17:17.429 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:17:17.429 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:17:17.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:17.429 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:17.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:17.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:18.409 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":732,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:18.430 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":732,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:18.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:18.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:19.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:19.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:20.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:20.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:21.431 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:21.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:21.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:21.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:22.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:22.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:23.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:23.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:24.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:24.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:25.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:25.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:26.425 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:26.448 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:26.448 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:17:26.448 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:26.448 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:17:26.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:26.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:27.398 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":738,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:27.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:27.430 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:17:27.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:27.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:28.433 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":738,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:28.463 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:28.463 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:17:28.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:28.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:29.489 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":738,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:29.518 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:17:29.519 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:29.519 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:17:29.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:29.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:30.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:30.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:30.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:17:31.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:31.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:31.617 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:17:31.618 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:31.618 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:32.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:32.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:32.588 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:32.619 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:32.619 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:17:32.619 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:17:32.619 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:32.619 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:17:33.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:33.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:33.594 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":738,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:33.616 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:33.616 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:17:34.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:34.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:34.614 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":738,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:34.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:34.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:17:35.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:35.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:35.619 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":738,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:35.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:17:35.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:17:35.642 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:17:35.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:17:35.643 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:17:35.643 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:17:35.677 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:17:35.677 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:17:35.678 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:35.678 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:36.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:36.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:36.650 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:36.681 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:37.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:37.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:38.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:38.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:39.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:39.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:39.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:39.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:40.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:40.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:40.686 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:40.708 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":738,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:41.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:41.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:42.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:42.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:43.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:43.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:43.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:43.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:17:44.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:44.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:45.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:45.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:46.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:46.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:47.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:47.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:48.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:48.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:49.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:49.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:49.681 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:49.714 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:17:49.715 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:17:49.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:49.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:17:50.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:50.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:50.744 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":740,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:50.769 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:17:50.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:17:51.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:51.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:51.763 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":740,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:17:51.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:17:52.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:52.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:53.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:53.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:54.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:54.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:55.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:55.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:56.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:56.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:57.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:57.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:58.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:58.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:17:59.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:17:59.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:00.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:00.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:01.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:01.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:02.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:02.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:03.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:03.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:04.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:04.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:05.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:05.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:06.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:06.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:07.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:07.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:08.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:08.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:09.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:09.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:10.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:10.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:11.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:11.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:12.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:12.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:13.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:13.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:14.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:14.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:15.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:15.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:16.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:16.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:17.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:17.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:18.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:18.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:19.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:19.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:20.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:20.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:21.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:21.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:22.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:22.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:23.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:23.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:24.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:24.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:25.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:25.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:26.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:26.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:27.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:27.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:28.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:28.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:29.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:29.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:30.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:30.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:31.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:31.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:32.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:32.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:33.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:33.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:34.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:34.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:35.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:35.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:36.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:36.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:37.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:37.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:38.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:38.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:39.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:39.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:40.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:40.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:41.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:41.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:42.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:42.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:43.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:43.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:44.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:44.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:45.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:45.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:46.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:46.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:47.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:47.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:48.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:48.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:49.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:49.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:50.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:50.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:51.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:51.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:52.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:52.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:53.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:53.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:54.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:54.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:55.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:55.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:56.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:56.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:57.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:57.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:58.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:58.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:18:59.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:18:59.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:00.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:00.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:01.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:01.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:02.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:02.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:03.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:03.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:04.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:04.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:05.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:05.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:06.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:06.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:07.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:07.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:08.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:08.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:09.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:09.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:10.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:10.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:11.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:11.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:12.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:12.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:13.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:13.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:14.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:14.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:15.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:15.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:16.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:16.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:17.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:17.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:18.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:18.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:19.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:19.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:20.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:20.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:21.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:21.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:21.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:19:21.787 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:19:21.787 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:19:21.788 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:19:21.788 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:19:21.793 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:19:21.794 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:19:21.794 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:21.794 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:22.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:22.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:22.807 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:22.829 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:23.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:23.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:24.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:24.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:25.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:25.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:25.830 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:25.830 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:26.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:26.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:26.824 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:26.845 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:27.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:27.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:28.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:28.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:29.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:29.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:29.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:29.846 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:30.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:30.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:30.889 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:30.919 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:31.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:31.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:32.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:32.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:33.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:33.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:33.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:33.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:34.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:34.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:34.939 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:34.962 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":740,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:35.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:35.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:36.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:36.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:37.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:37.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:37.963 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:37.963 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:38.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:38.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:38.966 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":744,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:38.998 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":744,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:38.998 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:19:38.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:38.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:19:39.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:39.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:39.989 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":744,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:19:40.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:40.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:19:40.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:40.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:41.035 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":744,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:19:41.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:41.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:19:41.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:41.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:42.045 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":744,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:19:42.065 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:19:42.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:42.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:19:42.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:42.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:43.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:19:43.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:43.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:44.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:19:44.110 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:44.110 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:19:44.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:44.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:45.030 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":744,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:45.062 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":744,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:19:45.062 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:19:45.063 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:19:45.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:45.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:19:45.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:45.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:46.076 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":744,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:19:46.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:46.106 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:19:46.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:46.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:47.112 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":744,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:19:47.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:19:47.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:19:47.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:47.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:48.112 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":744,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:19:48.130 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:19:48.130 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T05:19:48.130 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:19:48.131 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:19:48.131 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:19:48.131 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:19:48.150 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:19:48.151 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T05:19:48.151 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T05:19:48.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:19:48.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:48.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:49.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:49.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:50.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:50.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:51.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:51.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:52.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:52.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:53.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:53.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:54.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:54.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:55.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:55.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:56.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:56.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:57.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:57.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:58.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:58.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:19:59.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:19:59.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:00.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:00.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:01.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:01.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:02.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:02.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:03.152 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T05:20:03.152 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T05:20:03.153 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:20:03.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:20:03.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:20:03.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:03.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:04.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:04.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:05.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:05.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:06.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:06.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:07.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:07.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:08.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:08.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:08.800 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":748,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:20:08.823 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":748,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:20:08.823 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:20:08.823 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:20:08.823 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:20:09.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:09.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:09.796 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":748,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:20:09.827 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:20:09.827 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:20:10.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:10.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:10.834 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":748,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:20:10.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:20:11.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:11.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:12.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:12.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:13.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:13.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:14.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:14.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:15.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:15.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:16.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:16.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:17.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:17.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:18.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:18.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:19.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:19.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:20.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:20.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:21.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:21.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:22.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:22.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:23.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:23.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:24.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:24.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:25.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:25.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:26.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:26.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:27.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:27.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:28.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:28.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:29.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:29.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:30.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:30.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:31.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:31.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:32.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:32.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:33.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:33.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:34.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:34.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:35.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:35.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:36.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:36.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:37.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:37.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:38.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:38.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:39.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:39.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:40.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:40.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:41.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:41.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:42.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:42.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:43.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:43.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:44.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:44.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:45.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:45.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:46.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:46.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:47.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:47.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:48.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:48.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:49.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:49.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:50.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:50.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:51.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:51.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:52.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:52.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:53.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:53.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:54.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:54.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:55.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:55.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:56.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:56.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:57.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:57.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:58.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:58.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:20:59.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:20:59.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:00.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:00.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:01.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:01.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:02.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:02.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:03.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:03.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:04.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:04.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:05.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:05.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:06.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:06.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:07.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:07.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:08.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:08.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:09.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:09.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:10.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:10.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:11.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:11.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:12.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:12.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:13.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:13.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:14.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:14.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:15.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:15.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:16.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:16.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:17.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:17.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:18.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:18.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:19.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:19.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:20.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:20.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:21.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:21.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:22.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:22.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:23.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:23.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:24.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:24.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:25.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:25.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:26.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:26.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:27.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:27.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:28.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:28.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:29.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:29.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:30.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:30.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:31.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:31.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:32.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:32.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:33.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:33.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:34.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:34.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:35.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:35.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:36.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:36.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:37.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:37.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:38.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:38.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:39.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:39.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:40.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:40.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:40.858 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:21:40.858 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:21:40.858 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:21:40.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:40.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:21:40.865 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:21:40.865 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T05:21:40.865 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T05:21:40.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:21:41.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:41.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:42.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:42.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:43.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:43.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:44.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:44.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:45.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:45.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:46.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:46.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:47.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:47.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:48.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:48.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:49.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:49.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:50.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:50.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:51.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:51.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:52.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:52.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:53.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:53.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:54.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:54.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:55.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:55.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:55.866 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T05:21:55.866 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T05:21:55.867 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:21:55.867 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:55.867 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:21:56.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:56.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:56.868 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":750,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:21:56.889 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":750,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:21:56.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:21:56.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:56.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:21:57.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:57.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:57.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":750,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:21:57.895 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:57.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:21:58.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:58.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:58.907 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":750,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:21:58.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:58.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:21:59.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:21:59.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:21:59.915 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":750,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:21:59.945 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:21:59.946 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:21:59.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:22:00.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:00.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:01.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:22:01.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:01.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:02.087 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:22:02.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:02.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:22:02.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:02.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:03.128 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":750,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:22:03.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:03.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:03.152 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":750,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:22:03.152 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:22:03.153 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:22:03.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:03.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:22:04.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:04.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:04.125 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":750,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:22:04.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:04.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:22:05.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:05.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:05.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":750,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:22:05.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:05.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:22:06.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:06.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:06.190 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":750,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:22:06.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:22:06.220 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T05:22:06.220 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:22:06.220 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:22:06.220 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:22:06.220 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:22:06.241 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:22:06.241 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:22:06.242 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:22:06.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:22:07.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:07.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:08.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:08.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:09.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:09.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:10.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:10.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:11.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:11.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:12.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:12.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:13.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:13.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:14.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:14.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:15.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:15.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:16.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:16.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:17.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:17.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:18.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:18.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:19.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:19.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:20.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:20.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:21.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:21.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:21.243 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:22:21.243 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:22:21.244 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:22:21.244 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:21.244 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:22:22.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:22.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:23.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:23.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:24.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:24.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:25.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:25.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:26.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:26.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:26.926 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":752,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:22:26.948 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":752,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:22:26.949 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:22:26.949 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:26.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:22:27.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:27.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:27.946 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":752,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:22:27.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:22:27.978 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:22:28.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:28.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:28.974 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":752,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:22:28.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:22:29.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:29.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:30.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:30.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:31.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:31.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:32.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:32.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:33.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:33.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:34.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:34.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:35.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:35.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:36.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:36.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:37.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:37.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:38.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:38.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:39.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:39.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:40.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:40.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:41.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:41.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:42.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:42.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:43.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:43.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:44.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:44.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:45.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:45.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:46.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:46.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:47.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:47.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:48.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:48.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:49.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:49.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:50.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:50.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:51.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:51.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:52.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:52.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:53.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:53.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:54.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:54.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:55.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:55.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:56.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:56.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:57.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:57.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:58.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:58.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:22:59.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:22:59.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:00.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:00.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:01.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:01.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:02.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:02.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:03.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:03.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:04.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:04.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:05.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:05.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:06.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:06.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:07.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:07.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:08.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:08.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:09.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:09.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:10.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:10.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:11.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:11.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:12.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:12.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:13.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:13.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:14.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:14.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:15.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:15.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:16.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:16.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:17.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:17.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:18.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:18.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:19.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:19.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:20.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:20.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:21.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:21.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:22.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:22.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:23.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:23.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:23.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:23.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:24.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:24.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:25.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:25.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:26.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:26.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:27.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:27.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:28.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:28.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:29.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:29.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:30.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:30.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:31.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:31.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:32.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:32.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:33.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:33.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:34.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:34.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:35.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:35.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:36.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:36.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:37.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:37.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:39.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:39.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:40.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:40.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:41.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:41.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:42.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:42.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:42.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:42.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:44.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:43.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:44.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:44.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:46.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:46.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:46.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:46.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:48.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:48.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:49.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:49.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:50.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:50.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:51.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:51.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:52.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:52.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:53.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:53.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:54.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:54.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:55.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:54.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:55.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:55.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:56.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:56.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:57.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:57.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:58.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:58.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:23:58.997 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:23:58.997 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:23:58.997 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:23:58.997 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:23:58.998 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:23:59.008 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:23:59.008 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:23:59.008 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:23:59.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:23:59.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:23:59.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:00.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:00.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:01.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:01.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:02.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:02.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:03.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:03.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:04.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:04.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:05.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:05.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:06.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:06.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:07.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:07.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:08.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:08.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:09.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:09.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:10.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:10.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:11.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:11.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:12.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:12.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:13.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:13.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:14.009 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:24:14.009 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:24:14.009 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:24:14.010 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:14.010 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:24:14.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:14.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:15.060 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":760,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:15.081 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":760,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:15.081 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:24:15.081 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:15.081 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:24:16.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:16.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:16.080 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":760,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:16.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:16.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:24:17.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:17.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:17.091 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":760,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:17.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:17.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:24:18.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:18.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:18.119 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":760,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:18.142 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:24:18.143 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:18.143 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:24:19.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:19.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:19.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:24:20.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:20.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:20.197 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:24:20.197 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:20.198 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:24:21.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:21.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:21.192 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":760,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:21.213 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":760,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:21.214 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:24:21.214 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:24:21.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:21.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:24:22.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:22.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:22.199 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":760,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:22.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:22.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:24:23.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:23.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:23.247 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":760,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:23.293 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:23.293 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:24:24.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:24.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:24.318 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":760,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:24.339 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:24:24.340 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T05:24:24.340 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:24:24.340 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:24:24.340 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:24:24.340 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:24:24.358 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:24:24.358 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:24:24.359 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:24:24.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:24:25.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:25.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:26.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:26.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:27.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:27.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:28.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:28.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:29.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:29.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:30.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:30.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:31.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:31.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:32.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:32.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:33.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:33.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:34.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:34.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:35.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:34.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:36.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:35.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:36.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:36.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:37.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:37.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:38.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:38.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:39.361 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:24:39.361 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:24:39.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:24:39.362 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:39.362 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:24:39.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:39.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:40.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:40.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:41.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:41.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:42.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:42.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:43.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:43.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:44.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:44.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:45.012 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":762,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:45.034 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":762,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:24:45.034 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:24:45.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:45.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:24:45.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:45.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:46.027 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":762,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:46.057 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:24:46.057 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:24:46.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:46.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:47.065 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":762,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:24:47.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:24:47.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:47.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:48.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:48.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:49.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:49.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:50.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:50.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:51.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:51.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:52.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:52.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:53.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:53.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:54.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:54.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:55.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:55.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:56.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:56.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:57.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:57.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:58.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:58.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:24:59.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:24:59.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:00.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:00.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:01.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:01.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:02.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:02.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:03.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:03.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:04.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:04.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:06.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:06.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:07.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:07.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:08.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:08.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:09.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:09.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:10.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:10.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:11.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:11.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:12.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:12.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:13.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:13.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:14.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:14.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:15.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:15.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:16.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:16.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:17.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:17.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:18.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:18.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:19.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:19.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:20.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:20.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:21.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:21.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:22.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:22.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:23.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:23.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:24.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:24.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:25.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:25.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:26.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:26.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:27.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:27.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:28.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:28.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:29.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:29.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:30.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:30.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:31.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:31.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:32.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:32.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:33.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:33.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:34.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:34.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:35.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:35.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:36.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:36.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:37.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:37.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:38.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:38.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:39.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:39.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:40.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:40.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:41.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:41.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:42.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:42.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:43.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:43.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:44.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:44.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:45.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:45.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:46.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:46.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:47.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:47.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:48.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:48.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:49.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:49.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:50.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:50.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:51.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:51.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:52.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:52.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:53.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:53.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:54.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:54.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:55.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:55.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:56.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:56.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:57.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:57.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:58.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:58.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:25:59.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:25:59.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:00.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:00.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:01.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:01.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:02.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:02.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:03.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:03.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:04.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:04.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:05.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:05.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:06.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:06.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:07.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:07.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:08.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:08.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:09.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:09.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:10.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:10.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:10.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:10.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:11.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:11.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:12.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:12.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:13.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:13.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:14.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:14.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:15.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:15.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:16.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:16.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:17.098 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:26:17.098 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:26:17.098 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:26:17.098 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:26:17.099 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:26:17.105 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:26:17.105 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:26:17.106 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:26:17.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:26:17.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:17.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:18.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:18.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:19.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:19.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:20.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:20.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:21.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:21.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:22.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:22.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:23.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:23.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:24.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:24.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:25.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:25.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:26.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:26.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:27.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:27.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:28.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:28.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:29.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:29.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:30.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:30.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:31.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:31.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:32.107 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:26:32.108 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:26:32.108 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:26:32.108 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:32.108 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:26:32.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:32.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:33.163 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:33.192 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:33.193 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:26:33.193 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:33.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:26:33.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:33.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:34.148 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":770,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:34.179 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:34.179 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:26:34.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:34.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:35.147 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":770,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:35.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:35.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:26:35.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:35.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:36.200 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":770,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:36.223 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:26:36.223 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:36.223 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:26:36.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:36.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:37.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:26:37.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:37.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:38.324 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:26:38.325 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:38.325 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:26:38.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:38.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:39.310 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:39.335 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:39.335 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:26:39.335 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:26:39.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:39.335 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:26:39.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:39.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:40.335 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":770,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:40.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:40.353 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:26:40.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:40.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:41.339 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":770,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:41.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:41.370 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:26:41.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:41.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:42.352 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":770,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:42.371 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:26:42.371 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:26:42.372 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:26:42.372 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:26:42.372 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:26:42.372 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:26:42.421 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:26:42.421 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:26:42.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:42.422 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:26:42.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:42.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:43.410 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:43.429 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":770,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:43.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:43.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:44.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:44.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:45.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:45.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:46.430 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:46.430 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:26:46.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:46.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:47.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:47.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:48.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:48.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:49.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:49.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:50.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:50.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:51.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:51.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:52.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:52.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:53.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:53.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:54.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:54.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:55.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:55.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:56.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:56.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:57.319 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":774,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:57.339 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":774,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:26:57.339 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:26:57.339 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:57.340 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:26:57.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:57.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:58.314 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":774,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:26:58.336 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:26:58.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:26:58.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:58.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:26:59.347 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":774,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:26:59.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:26:59.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:26:59.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:00.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:00.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:01.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:01.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:02.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:02.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:03.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:03.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:04.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:04.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:05.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:05.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:06.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:06.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:07.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:07.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:08.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:08.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:09.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:09.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:10.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:10.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:11.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:11.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:12.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:12.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:13.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:13.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:14.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:14.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:15.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:15.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:16.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:16.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:17.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:17.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:18.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:18.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:19.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:19.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:20.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:20.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:21.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:21.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:22.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:22.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:23.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:23.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:24.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:24.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:25.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:25.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:26.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:26.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:27.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:27.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:28.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:28.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:29.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:29.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:30.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:30.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:31.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:31.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:32.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:32.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:33.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:33.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:34.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:34.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:35.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:35.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:36.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:36.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:37.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:37.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:38.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:38.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:39.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:39.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:40.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:40.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:41.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:41.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:42.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:42.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:43.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:43.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:44.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:44.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:45.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:45.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:46.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:46.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:47.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:47.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:48.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:48.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:49.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:49.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:50.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:50.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:51.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:51.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:52.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:52.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:53.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:53.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:54.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:54.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:55.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:55.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:56.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:56.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:57.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:57.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:58.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:58.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:27:59.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:27:59.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:00.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:00.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:01.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:01.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:02.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:02.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:03.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:03.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:04.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:04.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:05.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:05.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:06.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:06.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:07.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:07.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:08.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:08.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:09.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:09.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:10.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:10.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:11.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:11.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:12.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:12.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:13.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:13.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:14.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:14.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:15.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:15.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:16.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:16.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:17.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:17.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:18.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:18.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:19.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:19.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:20.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:20.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:21.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:21.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:22.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:22.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:23.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:23.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:24.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:24.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:25.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:25.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:26.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:26.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:27.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:27.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:28.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:28.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:29.378 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:28:29.378 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:28:29.378 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:28:29.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:29.379 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:28:29.384 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:28:29.384 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:28:29.384 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:29.385 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:29.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:29.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:30.370 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":774,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:30.392 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":774,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:30.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:30.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:31.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:31.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:32.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:32.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:33.394 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:33.394 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:33.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:33.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:34.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:34.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:35.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:35.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:36.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:36.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:37.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:37.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:38.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:38.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:38.908 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:38.940 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:38.940 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:28:38.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:38.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:28:39.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:39.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:39.905 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":782,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:28:39.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:39.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:28:40.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:40.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:40.957 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":782,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:28:40.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:40.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:28:41.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:41.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:41.989 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":782,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:28:42.011 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:28:42.011 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:42.011 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:28:42.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:42.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:43.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:28:43.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:43.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:44.084 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:28:44.084 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:44.084 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:44.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:44.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:45.075 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:45.103 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:45.104 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:28:45.104 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:28:45.104 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:45.104 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:28:45.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:45.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:46.137 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":782,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:28:46.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:46.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:28:46.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:46.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:47.157 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":782,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:28:47.179 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:47.180 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:28:47.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:47.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:48.206 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":782,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:28:48.237 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:28:48.238 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:28:48.238 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:28:48.238 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:28:48.238 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:28:48.238 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:28:48.254 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:28:48.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:28:48.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:48.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:48.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:48.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:49.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:49.290 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:49.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:49.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:50.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:50.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:51.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:51.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:52.291 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:52.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:52.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:52.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:53.289 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:53.312 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:53.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:53.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:54.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:54.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:55.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:55.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:56.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:28:56.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:28:56.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:56.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:57.307 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:57.329 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":782,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:28:57.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:57.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:58.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:58.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:28:59.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:28:59.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:00.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:29:00.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:29:00.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:00.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:01.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:01.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:02.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:02.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:03.550 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":786,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:29:03.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:03.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:03.582 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":786,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:29:03.582 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:29:03.582 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:29:03.583 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:29:04.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:04.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:04.583 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":786,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:29:04.609 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:29:04.610 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:29:05.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:05.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:05.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":786,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:29:05.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:29:06.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:06.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:07.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:07.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:08.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:08.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:09.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:09.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:10.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:10.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:11.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:11.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:12.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:12.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:13.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:13.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:14.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:14.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:15.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:15.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:16.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:16.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:17.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:17.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:18.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:18.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:19.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:19.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:20.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:20.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:21.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:21.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:22.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:22.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:23.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:23.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:24.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:24.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:25.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:25.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:26.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:26.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:27.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:27.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:28.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:28.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:29.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:29.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:30.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:30.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:31.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:31.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:32.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:32.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:33.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:33.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:34.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:34.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:35.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:35.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:36.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:36.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:37.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:37.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:38.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:38.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:39.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:39.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:40.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:40.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:41.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:41.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:42.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:42.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:43.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:43.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:44.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:44.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:45.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:45.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:46.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:46.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:47.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:47.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:48.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:48.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:49.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:49.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:50.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:50.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:51.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:51.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:52.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:52.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:53.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:53.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:54.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:54.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:55.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:55.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:56.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:56.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:57.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:57.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:58.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:58.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:29:59.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:29:59.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:00.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:00.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:01.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:01.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:02.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:02.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:03.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:03.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:04.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:04.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:05.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:05.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:06.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:06.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:07.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:07.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:08.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:08.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:09.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:09.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:10.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:10.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:11.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:11.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:12.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:12.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:13.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:13.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:14.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:14.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:15.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:15.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:16.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:16.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:17.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:17.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:18.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:18.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:19.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:19.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:20.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:20.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:21.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:21.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:22.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:22.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:23.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:23.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:24.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:24.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:25.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:25.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:26.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:26.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:27.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:27.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:28.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:28.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:29.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:29.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:30.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:30.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:31.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:31.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:32.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:32.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:33.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:33.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:34.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:34.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:35.637 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:30:35.637 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:30:35.637 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:30:35.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:35.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:30:35.644 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:30:35.644 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:30:35.645 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:35.645 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:30:35.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:35.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:36.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:36.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:36.661 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":786,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:36.683 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":786,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:37.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:37.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:38.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:38.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:39.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:39.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:39.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:39.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:30:40.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:40.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:41.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:41.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:42.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:42.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:43.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:43.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:44.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:44.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:45.232 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:45.340 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:45.340 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:30:45.340 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:45.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:30:45.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:45.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:46.313 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":794,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:30:46.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:46.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:30:46.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:46.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:47.335 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":794,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:30:47.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:47.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:30:47.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:47.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:48.359 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":794,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:30:48.364 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:30:48.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:48.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:30:48.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:48.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:49.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:30:49.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:49.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:50.494 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:30:50.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:50.494 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:30:50.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:50.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:51.506 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:51.524 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:51.524 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:30:51.524 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:30:51.525 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:51.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:30:51.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:51.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:52.502 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":794,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:30:52.534 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:52.534 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:30:52.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:52.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:53.546 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":794,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:30:53.576 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:53.577 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:30:53.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:53.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:54.576 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":794,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:30:54.605 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:30:54.605 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:30:54.605 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:30:54.605 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:30:54.606 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:30:54.606 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:30:54.624 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:30:54.624 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:30:54.625 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:54.625 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:30:54.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:54.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:55.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:55.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:55.692 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:55.714 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:56.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:56.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:57.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:57.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:58.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:58.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:58.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:30:58.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:30:59.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:30:59.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:30:59.693 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:30:59.716 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:31:00.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:00.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:01.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:01.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:02.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:02.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:02.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:31:02.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:31:03.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:03.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:03.698 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:31:03.720 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":794,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:31:04.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:04.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:05.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:05.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:06.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:06.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:06.720 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:31:06.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:31:07.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:07.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:08.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:08.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:09.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:09.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:09.596 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":798,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:31:09.618 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":798,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:31:09.619 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:31:09.619 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:31:09.619 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:31:10.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:10.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:10.615 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":798,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:31:10.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:31:10.646 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:31:11.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:11.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:11.647 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":798,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:31:11.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:31:12.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:12.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:13.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:13.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:14.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:14.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:15.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:15.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:16.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:16.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:17.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:17.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:18.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:18.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:19.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:19.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:20.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:20.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:21.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:21.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:22.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:22.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:23.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:23.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:24.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:24.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:25.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:25.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:26.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:26.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:27.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:27.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:28.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:28.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:29.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:29.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:30.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:30.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:31.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:31.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:32.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:32.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:33.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:33.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:34.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:34.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:35.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:35.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:36.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:36.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:37.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:37.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:38.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:38.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:39.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:39.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:40.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:40.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:41.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:41.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:42.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:42.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:43.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:43.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:44.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:44.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:45.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:45.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:46.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:46.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:47.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:47.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:48.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:48.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:49.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:49.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:50.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:50.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:51.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:51.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:52.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:52.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:53.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:53.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:54.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:54.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:55.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:55.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:56.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:56.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:57.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:57.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:58.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:58.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:31:59.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:31:59.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:00.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:00.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:01.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:01.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:02.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:02.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:03.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:03.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:04.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:04.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:05.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:05.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:06.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:06.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:07.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:07.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:08.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:08.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:09.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:09.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:10.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:10.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:11.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:11.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:12.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:12.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:13.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:13.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:14.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:14.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:15.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:15.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:16.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:16.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:17.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:17.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:18.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:18.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:19.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:19.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:20.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:20.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:21.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:21.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:23.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:23.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:24.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:24.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:25.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:25.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:26.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:26.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:27.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:27.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:28.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:28.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:29.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:29.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:30.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:30.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:31.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:31.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:32.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:32.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:33.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:33.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:34.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:34.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:35.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:35.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:36.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:36.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:37.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:37.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:38.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:37.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:38.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:38.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:39.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:39.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:40.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:40.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:41.673 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:32:41.673 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:32:41.673 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:32:41.674 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:41.674 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:32:41.681 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:32:41.681 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:32:41.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:41.682 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:32:41.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:41.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:42.670 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":798,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:42.699 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":798,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:42.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:42.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:43.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:43.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:44.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:44.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:45.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:45.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:32:45.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:45.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:46.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:46.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:47.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:47.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:48.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:48.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:49.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:49.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:50.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:50.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:51.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:51.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:52.931 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:52.954 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:52.954 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:32:52.954 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:52.954 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:32:52.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:52.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:53.922 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":806,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:32:53.944 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:53.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:32:53.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:53.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:54.987 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":806,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:32:55.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:55.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:55.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:55.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:32:55.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:55.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:56.024 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":806,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:32:56.046 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:32:56.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:56.046 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:32:56.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:56.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:57.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:32:57.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:57.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:58.150 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:32:58.150 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:58.151 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:32:58.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:58.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:32:59.121 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:59.153 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:32:59.153 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:32:59.153 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:32:59.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:32:59.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:32:59.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:32:59.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:00.123 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":806,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:33:00.152 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:00.152 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:33:00.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:00.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:01.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":806,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:33:01.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:01.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:33:01.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:01.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:02.158 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":806,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:33:02.180 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:33:02.180 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:33:02.181 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:33:02.181 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:33:02.181 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:33:02.181 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:33:02.199 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:33:02.199 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:33:02.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:02.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:33:03.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:03.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:03.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:03.296 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:03.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:03.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:05.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:05.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:06.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:06.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:06.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:06.297 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:33:07.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:07.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:07.308 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:07.330 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:08.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:08.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:09.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:09.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:10.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:10.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:10.331 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:10.331 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:33:11.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:11.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:11.306 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:11.329 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":806,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:12.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:12.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:13.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:13.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:14.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:14.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:14.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:14.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:33:15.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:15.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:15.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:15.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:16.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:16.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:17.381 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:17.403 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:33:17.403 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:33:17.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:17.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:33:17.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:17.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:18.394 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":810,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:33:18.425 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:33:18.426 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:33:18.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:18.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:19.382 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":810,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:33:19.404 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:33:19.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:19.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:20.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:20.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:21.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:21.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:22.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:22.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:23.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:23.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:24.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:24.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:25.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:25.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:26.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:26.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:28.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:28.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:28.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:28.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:29.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:29.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:30.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:30.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:31.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:31.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:33.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:33.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:34.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:34.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:34.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:34.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:35.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:35.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:36.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:36.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:37.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:37.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:39.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:39.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:40.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:40.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:41.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:41.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:42.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:42.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:42.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:42.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:44.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:44.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:45.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:45.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:46.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:46.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:47.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:47.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:48.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:48.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:49.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:49.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:50.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:50.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:51.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:51.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:52.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:52.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:53.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:53.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:54.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:54.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:55.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:55.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:56.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:56.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:57.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:57.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:58.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:58.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:33:59.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:33:59.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:00.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:00.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:01.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:01.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:01.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:01.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:02.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:02.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:03.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:03.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:04.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:04.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:05.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:05.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:06.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:06.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:07.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:07.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:08.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:08.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:09.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:09.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:10.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:10.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:11.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:11.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:12.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:12.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:13.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:13.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:14.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:14.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:15.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:15.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:16.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:16.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:17.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:17.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:18.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:18.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:19.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:19.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:20.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:20.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:21.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:21.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:22.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:22.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:23.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:23.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:24.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:24.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:25.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:25.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:26.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:26.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:27.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:27.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:28.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:28.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:29.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:29.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:30.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:30.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:31.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:31.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:32.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:32.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:33.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:33.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:34.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:34.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:35.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:35.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:36.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:36.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:37.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:37.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:38.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:38.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:39.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:39.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:40.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:40.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:41.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:41.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:42.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:42.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:43.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:43.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:44.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:44.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:45.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:45.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:46.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:46.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:47.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:47.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:48.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:48.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:49.405 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:34:49.405 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:34:49.405 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:34:49.406 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:34:49.406 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:34:49.411 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:34:49.411 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:34:49.412 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:34:49.412 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:34:49.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:49.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:50.381 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:34:50.403 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:34:50.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:50.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:51.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:51.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:52.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:52.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:53.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:34:53.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:34:53.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:53.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:54.391 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:34:54.414 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":810,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:34:54.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:54.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:55.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:55.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:56.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:56.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:57.415 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:34:57.415 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:34:57.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:57.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:58.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:58.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:34:59.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:34:59.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:00.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:00.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:01.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:01.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:02.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:02.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:03.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:03.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:04.629 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:04.651 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:04.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:35:04.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:04.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:35:04.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:04.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:05.660 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":818,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:05.682 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:05.682 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:35:05.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:05.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:06.671 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":818,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:06.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:06.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:35:06.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:06.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:07.710 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":818,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:35:07.734 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:35:07.734 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:07.734 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:35:07.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:07.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:08.824 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:35:08.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:08.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:09.824 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:35:09.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:09.825 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:35:09.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:09.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:10.824 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:10.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:10.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:10.854 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:10.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:35:10.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:35:10.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:10.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:35:11.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:11.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:11.934 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":818,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:11.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:11.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:35:12.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:12.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:12.951 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":818,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:12.973 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:12.974 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:35:13.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:13.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:13.970 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":818,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:35:13.993 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:35:13.993 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:35:13.993 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:35:13.993 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:35:13.994 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:35:13.994 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:35:14.023 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:35:14.023 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:35:14.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:14.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:35:14.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:14.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:15.038 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:15.061 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:15.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:15.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:16.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:16.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:17.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:17.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:18.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:18.062 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:35:18.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:18.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:19.056 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:19.087 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:19.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:19.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:20.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:20.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:21.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:21.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:22.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:22.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:35:22.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:22.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:23.069 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:23.092 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":818,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:23.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:23.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:24.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:24.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:25.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:25.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:26.093 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:26.093 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:35:26.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:26.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:27.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:27.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:28.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:28.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:29.590 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":822,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:29.612 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":822,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:35:29.612 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:35:29.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:29.613 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:35:29.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:29.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:30.591 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":822,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:30.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:35:30.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:35:30.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:30.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:31.623 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":822,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:35:31.653 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:35:31.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:31.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:32.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:32.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:33.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:33.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:34.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:34.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:35.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:35.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:36.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:36.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:37.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:37.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:38.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:38.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:39.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:39.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:40.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:40.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:41.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:41.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:42.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:42.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:43.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:43.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:44.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:44.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:45.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:45.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:46.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:46.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:47.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:47.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:48.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:48.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:49.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:49.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:50.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:50.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:51.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:51.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:52.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:52.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:53.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:53.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:54.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:54.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:55.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:55.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:56.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:56.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:57.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:57.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:58.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:58.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:35:59.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:35:59.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:00.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:00.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:01.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:01.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:02.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:02.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:03.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:03.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:04.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:04.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:05.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:05.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:06.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:06.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:07.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:07.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:08.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:08.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:09.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:09.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:10.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:10.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:11.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:11.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:12.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:12.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:13.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:13.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:14.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:14.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:15.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:15.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:16.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:16.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:17.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:17.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:18.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:18.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:19.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:19.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:20.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:20.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:21.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:21.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:22.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:22.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:23.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:23.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:24.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:24.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:25.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:25.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:26.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:26.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:27.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:27.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:28.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:28.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:29.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:29.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:30.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:30.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:31.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:31.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:32.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:32.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:33.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:33.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:34.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:34.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:35.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:35.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:36.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:36.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:37.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:37.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:38.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:38.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:39.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:39.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:40.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:40.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:41.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:41.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:42.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:42.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:43.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:43.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:44.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:44.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:45.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:45.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:46.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:46.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:47.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:47.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:48.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:48.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:49.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:49.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:50.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:50.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:51.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:51.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:52.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:52.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:53.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:53.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:54.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:54.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:55.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:55.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:56.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:56.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:57.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:57.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:58.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:58.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:36:59.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:36:59.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:00.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:00.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:01.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:01.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:01.654 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:37:01.654 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:37:01.654 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:37:01.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:01.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:37:01.661 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:37:01.661 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:37:01.661 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:01.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:37:02.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:02.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:02.689 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":822,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:02.718 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":822,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:03.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:03.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:04.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:04.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:05.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:05.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:05.719 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:05.719 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:37:06.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:06.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:07.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:07.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:08.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:08.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:09.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:09.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:10.547 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":828,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:10.567 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":828,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:10.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:37:10.568 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:10.568 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:37:10.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:10.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:11.564 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":828,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:11.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:11.586 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:37:11.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:11.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:12.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:12.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:12.803 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":828,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:12.821 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:12.821 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:37:13.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:13.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:13.821 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":828,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:13.844 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:37:13.844 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:13.844 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:37:14.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:14.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:14.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:37:15.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:15.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:15.962 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:37:15.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:15.962 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:37:16.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:16.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:16.954 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":828,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:16.984 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":828,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:16.984 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:37:16.984 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:37:16.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:16.985 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:37:17.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:17.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:17.997 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":828,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:18.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:18.030 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:37:18.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:18.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:19.043 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":828,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:19.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:19.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:37:19.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:19.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:20.089 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":828,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:20.112 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:37:20.112 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T05:37:20.112 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:37:20.112 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:37:20.112 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:37:20.113 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:37:20.132 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:37:20.132 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:37:20.132 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:37:20.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:37:20.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:20.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:21.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:21.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:22.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:22.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:23.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:23.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:24.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:24.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:25.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:25.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:26.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:26.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:27.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:27.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:28.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:28.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:29.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:29.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:30.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:30.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:31.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:31.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:32.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:32.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:33.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:33.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:34.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:34.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:35.134 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:37:35.134 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:37:35.134 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:37:35.135 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:35.135 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:37:35.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:35.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:36.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:36.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:37.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:37.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:38.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:38.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:39.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:39.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:40.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:40.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:40.791 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":830,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:40.814 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":830,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:37:40.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:37:40.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:40.815 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:37:41.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:41.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:41.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":830,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:41.849 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:37:41.849 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:37:42.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:42.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:42.855 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":830,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:37:42.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:37:43.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:43.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:44.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:44.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:45.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:45.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:46.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:46.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:47.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:47.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:48.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:48.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:49.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:49.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:50.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:50.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:51.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:51.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:52.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:52.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:53.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:53.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:54.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:54.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:55.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:55.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:56.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:56.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:57.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:57.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:58.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:58.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:37:59.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:37:59.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:00.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:00.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:01.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:01.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:02.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:02.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:03.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:03.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:04.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:04.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:05.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:05.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:06.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:06.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:07.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:07.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:08.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:08.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:09.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:09.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:10.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:10.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:11.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:11.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:12.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:12.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:13.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:13.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:14.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:14.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:15.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:15.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:16.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:16.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:17.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:17.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:18.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:18.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:19.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:19.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:20.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:20.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:21.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:21.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:22.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:22.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:23.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:23.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:24.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:24.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:25.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:25.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:26.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:26.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:27.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:27.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:28.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:28.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:29.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:29.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:30.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:30.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:31.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:31.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:32.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:32.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:33.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:33.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:34.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:34.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:35.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:35.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:36.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:36.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:37.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:37.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:38.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:38.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:39.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:39.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:40.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:40.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:41.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:41.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:42.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:42.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:43.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:43.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:44.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:44.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:45.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:45.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:46.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:46.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:47.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:47.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:48.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:48.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:49.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:49.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:50.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:50.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:51.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:51.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:52.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:52.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:53.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:53.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:54.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:54.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:55.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:55.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:56.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:56.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:57.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:57.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:58.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:58.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:38:59.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:38:59.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:00.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:00.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:01.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:01.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:02.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:02.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:03.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:03.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:04.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:04.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:05.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:05.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:06.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:06.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:07.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:07.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:08.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:08.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:09.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:09.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:10.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:10.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:11.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:11.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:12.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:12.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:12.877 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:39:12.877 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:39:12.877 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:39:12.878 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:12.878 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:39:12.884 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:39:12.884 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:39:12.885 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:39:12.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:39:13.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:13.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:14.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:14.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:15.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:15.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:16.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:16.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:17.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:17.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:18.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:18.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:19.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:19.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:20.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:20.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:21.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:21.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:22.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:22.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:23.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:23.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:24.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:24.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:25.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:25.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:26.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:26.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:27.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:27.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:27.886 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:39:27.886 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:39:27.886 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:39:27.887 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:27.887 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:39:28.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:28.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:28.901 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:39:28.924 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:39:28.924 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:39:28.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:28.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:39:29.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:29.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:29.937 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":838,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:29.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:29.969 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:39:30.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:30.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:31.066 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":838,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:31.087 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:31.087 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:39:31.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:31.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:32.066 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":838,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:32.088 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:39:32.089 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:32.089 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:39:32.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:32.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:33.182 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:39:33.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:33.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:34.183 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:39:34.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:34.184 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:39:34.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:34.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:35.168 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:39:35.199 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:39:35.200 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:39:35.200 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:39:35.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:35.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:39:35.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:35.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:36.206 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":838,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:36.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:36.237 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:39:36.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:36.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:37.287 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":838,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:37.308 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:37.309 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:39:37.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:37.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:38.330 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":838,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:39:38.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:39:38.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T05:39:38.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:39:38.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:39:38.352 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:39:38.352 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:39:38.375 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:39:38.375 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:39:38.376 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:39:38.376 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:39:38.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:38.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:39.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:39.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:40.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:40.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:41.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:41.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:42.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:42.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:43.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:43.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:44.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:44.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:45.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:45.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:46.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:46.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:47.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:47.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:48.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:48.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:49.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:49.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:50.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:50.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:51.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:51.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:52.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:52.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:53.377 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:39:53.377 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:39:53.378 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:39:53.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:39:53.378 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:39:53.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:53.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:54.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:54.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:55.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:55.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:56.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:56.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:57.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:57.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:58.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:58.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:39:59.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:39:59.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:00.590 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":840,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:40:00.613 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":840,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:40:00.613 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:40:00.613 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:40:00.613 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:40:00.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:00.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:01.634 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":840,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:40:01.660 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:40:01.660 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:40:01.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:01.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:02.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:02.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:02.676 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":840,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:40:02.698 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:40:03.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:03.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:04.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:04.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:05.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:05.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:06.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:06.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:07.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:07.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:08.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:08.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:09.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:09.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:10.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:10.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:11.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:11.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:12.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:12.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:13.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:13.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:14.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:14.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:15.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:15.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:16.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:16.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:17.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:17.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:18.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:18.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:19.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:19.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:20.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:20.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:21.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:21.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:22.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:22.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:23.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:23.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:24.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:24.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:25.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:25.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:26.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:26.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:27.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:27.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:28.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:28.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:29.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:29.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:30.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:30.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:31.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:31.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:32.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:32.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:33.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:33.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:34.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:34.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:35.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:35.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:36.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:36.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:37.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:37.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:38.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:38.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:39.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:39.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:40.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:40.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:41.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:41.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:42.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:42.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:43.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:43.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:44.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:44.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:45.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:45.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:46.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:46.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:47.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:47.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:48.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:48.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:49.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:49.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:50.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:50.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:51.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:51.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:52.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:52.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:53.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:53.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:54.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:54.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:55.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:55.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:56.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:56.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:57.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:57.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:58.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:58.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:40:59.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:40:59.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:00.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:00.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:01.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:01.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:02.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:02.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:03.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:03.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:04.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:04.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:05.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:05.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:06.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:06.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:07.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:07.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:08.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:08.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:09.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:09.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:10.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:10.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:11.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:11.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:12.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:12.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:13.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:13.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:14.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:14.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:15.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:15.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:16.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:16.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:17.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:17.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:18.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:18.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:19.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:19.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:20.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:20.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:21.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:21.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:22.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:22.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:23.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:23.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:24.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:24.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:25.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:25.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:26.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:26.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:27.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:27.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:28.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:28.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:29.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:29.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:30.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:30.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:31.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:31.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:32.699 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:41:32.700 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:41:32.700 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:41:32.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:32.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:41:32.705 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:41:32.705 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T05:41:32.706 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T05:41:32.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:41:32.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:32.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:33.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:33.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:34.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:34.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:35.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:35.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:36.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:36.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:37.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:37.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:38.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:38.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:39.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:39.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:40.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:40.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:41.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:41.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:42.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:42.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:43.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:43.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:44.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:44.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:45.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:45.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:46.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:46.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:47.707 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T05:41:47.707 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T05:41:47.707 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:41:47.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:47.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:41:47.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:47.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:48.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:48.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:48.793 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:48.815 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:48.816 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:41:48.816 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:48.816 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:41:49.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":844,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:49.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:49.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:41:49.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:49.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:50.664 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":844,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:50.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:50.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:41:50.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:50.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:51.680 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":844,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:51.702 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:41:51.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:51.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:41:51.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:51.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:52.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:41:52.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:52.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:53.807 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:41:53.808 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:53.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:41:53.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:53.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:54.814 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:54.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:54.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:54.837 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:54.837 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:41:54.837 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:41:54.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:54.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:41:55.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:55.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:55.856 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":844,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:55.886 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:55.887 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:41:56.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:56.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:56.878 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":844,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:56.899 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:56.900 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:41:57.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:57.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:57.930 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":844,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:41:57.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:41:57.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:41:57.960 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:41:57.961 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:41:57.961 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:41:57.961 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:41:57.980 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:41:57.980 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:41:57.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:41:57.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:41:58.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:58.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:41:59.047 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:59.076 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:41:59.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:41:59.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:00.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:00.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:01.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:01.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:02.078 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:42:02.078 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:42:02.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:02.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:03.073 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:03.096 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:03.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:03.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:04.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:04.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:05.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:05.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:06.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:42:06.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:42:06.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:06.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:07.097 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:07.127 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":844,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:07.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:07.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:08.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:08.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:09.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:09.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:10.128 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:42:10.128 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:42:10.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:10.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:11.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:11.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:12.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:12.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:13.585 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":848,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:13.607 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":848,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:42:13.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:42:13.608 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:42:13.608 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:42:13.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:13.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:14.610 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":848,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:42:14.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:42:14.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:42:14.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:14.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:15.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":848,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:42:15.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:42:15.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:15.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:16.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:16.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:17.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:17.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:18.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:18.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:19.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:19.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:20.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:20.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:21.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:21.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:22.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:22.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:23.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:23.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:24.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:24.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:25.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:25.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:26.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:26.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:27.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:27.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:28.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:28.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:29.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:29.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:30.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:30.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:31.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:31.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:32.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:32.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:33.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:33.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:34.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:34.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:35.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:35.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:36.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:36.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:37.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:37.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:38.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:38.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:39.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:39.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:40.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:40.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:41.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:41.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:42.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:42.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:43.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:43.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:44.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:44.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:45.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:45.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:46.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:46.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:47.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:47.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:48.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:48.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:49.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:49.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:50.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:50.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:51.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:51.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:52.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:52.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:53.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:53.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:54.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:54.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:55.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:55.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:56.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:56.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:57.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:57.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:58.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:58.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:42:59.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:42:59.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:00.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:00.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:01.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:01.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:02.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:02.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:03.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:03.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:04.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:04.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:05.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:05.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:06.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:06.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:07.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:07.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:08.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:08.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:09.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:09.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:10.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:10.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:11.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:11.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:12.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:12.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:13.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:13.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:14.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:14.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:15.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:15.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:16.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:16.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:17.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:17.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:18.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:18.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:19.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:19.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:20.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:20.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:21.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:21.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:22.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:22.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:23.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:23.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:24.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:24.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:25.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:25.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:26.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:26.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:27.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:27.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:28.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:28.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:29.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:29.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:30.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:30.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:31.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:31.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:33.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:33.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:34.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:34.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:35.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:35.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:36.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:36.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:37.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:37.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:38.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:38.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:39.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:39.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:40.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:40.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:41.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:41.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:42.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:42.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:43.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:43.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:44.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:44.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:45.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:45.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:45.672 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:43:45.672 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T05:43:45.672 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T05:43:45.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:45.673 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T05:43:45.678 INFO:tasks.ceph.mon.c:Started 2019-10-01T05:43:45.679 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:43:45.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:45.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:43:46.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:46.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:46.627 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":848,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:46.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":848,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:47.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:47.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:48.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:48.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:49.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:49.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:49.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:49.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:43:50.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:50.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:50.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:50.679 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:50.679 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:43:50.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:50.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:43:51.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:51.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:51.659 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":850,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:51.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:51.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:43:52.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:52.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:52.663 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":850,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:52.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:52.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:43:53.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:53.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:53.688 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":850,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:53.718 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:43:53.718 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:53.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:43:54.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:54.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:54.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:43:55.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:55.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:55.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:43:55.845 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:55.845 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:43:56.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:56.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:56.846 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:56.876 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:43:56.876 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:43:56.876 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:43:56.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:56.877 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:43:57.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:57.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:57.898 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":850,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:57.929 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:57.929 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:43:58.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:58.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:58.930 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":850,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:58.961 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:43:58.962 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:43:59.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:43:59.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:43:59.966 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":850,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:43:59.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:43:59.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T05:43:59.996 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:43:59.997 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:43:59.997 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:43:59.997 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:44:00.031 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:44:00.032 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:44:00.032 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:44:00.032 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:44:00.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:00.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:01.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:01.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:02.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:02.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:03.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:03.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:04.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:04.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:05.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:05.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:06.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:06.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:07.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:07.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:08.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:08.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:09.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:09.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:10.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:10.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:11.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:11.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:12.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:12.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:13.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:13.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:14.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:14.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:15.033 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:44:15.034 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:44:15.034 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:44:15.034 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:44:15.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:44:15.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:15.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:16.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:16.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:17.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:17.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:18.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:18.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:19.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:19.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:20.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:20.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:20.685 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":852,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:44:20.708 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":852,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:44:20.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:44:20.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:44:20.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:44:21.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:21.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:21.658 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":852,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:44:21.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:44:21.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:44:22.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:22.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:22.689 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":852,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:44:22.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:44:23.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:23.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:24.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:24.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:25.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:25.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:26.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:26.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:27.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:27.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:28.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:28.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:29.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:29.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:30.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:30.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:31.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:31.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:32.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:32.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:33.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:33.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:34.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:34.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:35.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:35.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:36.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:36.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:37.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:37.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:38.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:38.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:39.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:39.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:40.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:40.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:41.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:41.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:42.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:42.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:43.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:43.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:44.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:44.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:45.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:45.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:46.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:46.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:47.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:47.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:48.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:48.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:49.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:49.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:50.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:50.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:51.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:51.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:52.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:52.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:53.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:53.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:54.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:54.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:55.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:55.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:56.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:56.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:57.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:57.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:58.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:58.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:44:59.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:44:59.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:00.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:00.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:01.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:01.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:02.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:02.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:03.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:03.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:04.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:04.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:05.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:05.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:06.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:06.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:07.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:07.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:08.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:08.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:09.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:09.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:10.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:10.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:11.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:11.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:12.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:12.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:13.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:13.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:14.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:14.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:15.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:15.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:16.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:16.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:17.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:17.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:18.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:18.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:19.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:19.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:20.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:20.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:21.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:21.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:22.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:22.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:23.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:23.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:24.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:24.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:25.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:25.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:26.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:26.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:27.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:27.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:28.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:28.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:29.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:29.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:30.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:30.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:31.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:31.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:32.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:32.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:33.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:33.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:34.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:34.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:35.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:35.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:36.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:36.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:37.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:37.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:38.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:38.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:39.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:39.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:40.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:40.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:41.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:41.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:42.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:42.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:43.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:43.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:44.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:44.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:45.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:45.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:46.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:46.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:47.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:47.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:48.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:48.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:49.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:49.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:50.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:50.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:51.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:51.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:52.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:52.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:52.712 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:45:52.712 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:45:52.712 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:45:52.712 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:45:52.713 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:45:52.720 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:45:52.720 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:45:52.721 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:45:52.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:45:53.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:53.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:54.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:54.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:55.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:55.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:56.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:56.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:57.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:57.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:58.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:58.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:45:59.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:45:59.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:00.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:00.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:01.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:01.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:02.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:02.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:03.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:03.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:04.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:04.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:05.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:05.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:06.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:06.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:07.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:07.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:07.722 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:46:07.722 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:46:07.722 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:46:07.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:07.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:46:08.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:08.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:08.711 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:08.734 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:08.734 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:46:08.734 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:08.735 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:46:09.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:09.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:09.711 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":862,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:09.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:09.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:46:10.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:10.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:10.738 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":862,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:10.770 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:10.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:46:11.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:11.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:11.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":862,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:11.795 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:46:11.796 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:11.796 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:46:12.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:12.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:12.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:46:13.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:13.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:13.910 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:46:13.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:13.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:46:14.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:14.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:14.900 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:14.922 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:14.922 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:46:14.922 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:46:14.922 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:14.922 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:46:15.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:15.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:15.907 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":862,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:15.930 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:15.930 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:46:16.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:16.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:16.905 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":862,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:16.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:16.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:46:17.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:17.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:17.951 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":862,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:17.981 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:46:17.981 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:46:17.981 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:46:17.981 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:46:17.981 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:46:17.981 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:46:18.001 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:46:18.001 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:46:18.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:18.002 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:46:18.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:18.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:19.065 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:19.097 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:19.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:19.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:20.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:20.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:21.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:21.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:22.098 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:22.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:46:22.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:22.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:23.108 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:23.130 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":862,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:23.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:23.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:24.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:24.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:25.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:25.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:26.131 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:26.131 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:46:26.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:26.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:27.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:27.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:28.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:28.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:29.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:29.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:30.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:30.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:31.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:31.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:32.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:32.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:32.750 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:32.772 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:46:32.772 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:46:32.773 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:32.773 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:46:33.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:33.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:33.728 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":866,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:33.750 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:46:33.750 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:46:34.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:34.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:34.704 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":866,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:46:34.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:46:35.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:35.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:36.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:36.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:37.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:37.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:38.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:38.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:39.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:39.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:40.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:40.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:41.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:41.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:42.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:42.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:43.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:43.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:44.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:44.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:45.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:45.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:46.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:46.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:47.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:47.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:48.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:48.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:49.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:49.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:50.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:50.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:51.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:51.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:52.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:52.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:53.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:53.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:54.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:54.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:55.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:55.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:56.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:56.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:57.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:57.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:58.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:58.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:46:59.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:46:59.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:00.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:00.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:01.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:01.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:02.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:02.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:03.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:03.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:04.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:04.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:05.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:05.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:06.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:06.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:07.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:07.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:08.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:08.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:09.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:09.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:10.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:10.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:11.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:11.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:12.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:12.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:13.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:13.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:14.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:14.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:15.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:15.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:16.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:16.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:17.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:17.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:18.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:18.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:19.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:19.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:20.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:20.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:21.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:21.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:22.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:22.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:23.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:23.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:24.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:24.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:25.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:25.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:26.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:26.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:27.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:27.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:28.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:28.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:29.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:29.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:30.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:30.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:31.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:31.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:32.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:32.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:33.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:33.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:34.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:34.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:35.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:35.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:36.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:36.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:37.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:37.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:38.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:38.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:39.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:39.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:40.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:40.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:41.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:41.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:42.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:42.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:43.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:43.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:44.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:44.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:45.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:45.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:46.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:46.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:47.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:47.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:48.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:48.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:49.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:49.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:50.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:50.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:51.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:51.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:52.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:52.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:53.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:53.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:54.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:54.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:55.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:55.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:56.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:56.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:57.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:57.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:58.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:58.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:47:59.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:47:59.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:00.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:00.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:01.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:01.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:02.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:02.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:03.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:03.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:04.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:04.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:04.728 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:48:04.728 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:48:04.728 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:48:04.728 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:04.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:48:04.734 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:48:04.734 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:48:04.735 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:04.735 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:05.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:05.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:05.711 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:05.740 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:06.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:06.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:07.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:07.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:08.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:08.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:08.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:08.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:09.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:09.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:09.746 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:09.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:10.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:10.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:11.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:11.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:12.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:12.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:12.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:12.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:13.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:13.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:13.796 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:13.819 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":866,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:14.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:14.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:15.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:15.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:16.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:16.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:16.821 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:16.821 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:17.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:17.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:17.849 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:17.871 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:17.872 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:48:17.872 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:17.872 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:48:18.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:18.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:18.832 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":868,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:18.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:18.860 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:48:19.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:19.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:19.874 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":868,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:48:19.906 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:19.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:48:20.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:20.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:20.897 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":868,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:20.926 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:48:20.927 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:20.927 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:48:21.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:21.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:22.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:48:22.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:22.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:23.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:48:23.075 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:23.075 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:23.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:23.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:24.064 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:24.094 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:24.094 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:48:24.094 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:48:24.095 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:24.095 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:48:24.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:24.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:25.099 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":868,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:25.131 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:25.131 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:48:25.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:25.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:26.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":868,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:48:26.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:26.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:48:26.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:26.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:27.119 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":868,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:27.150 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:48:27.150 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:48:27.150 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:48:27.150 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:48:27.151 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:48:27.151 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:48:27.175 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:48:27.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:48:27.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:27.175 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:27.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:27.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:28.169 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:28.201 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:28.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:28.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:29.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:29.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:30.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:30.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:31.202 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:31.202 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:31.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:31.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:32.174 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:32.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:32.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:32.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:33.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:33.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:34.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:34.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:35.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:35.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:35.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:35.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:36.185 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:36.207 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":868,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:36.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:36.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:37.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:37.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:38.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:38.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:39.208 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:39.208 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:48:39.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:39.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:40.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:40.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:41.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:41.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:42.342 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":870,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:42.364 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":870,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:48:42.364 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:48:42.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:42.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:48:42.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:42.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:43.368 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":870,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:43.398 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:48:43.399 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:48:43.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:43.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:44.394 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":870,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:48:44.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:48:44.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:44.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:45.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:45.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:46.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:46.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:47.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:47.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:48.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:48.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:49.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:49.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:50.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:50.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:51.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:51.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:52.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:52.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:53.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:53.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:54.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:54.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:55.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:55.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:56.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:56.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:57.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:57.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:58.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:58.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:48:59.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:48:59.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:00.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:00.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:01.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:01.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:02.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:02.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:03.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:03.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:04.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:04.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:05.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:05.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:06.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:06.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:07.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:07.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:08.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:08.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:09.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:09.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:10.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:10.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:11.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:11.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:12.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:12.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:13.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:13.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:14.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:14.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:15.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:15.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:16.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:16.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:17.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:17.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:18.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:18.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:19.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:19.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:20.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:20.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:21.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:21.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:22.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:22.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:23.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:23.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:24.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:24.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:25.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:25.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:26.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:26.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:27.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:27.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:28.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:28.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:29.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:29.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:30.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:30.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:31.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:31.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:32.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:32.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:33.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:33.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:34.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:34.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:35.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:35.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:36.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:36.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:37.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:37.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:38.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:38.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:39.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:39.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:40.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:40.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:41.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:41.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:42.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:42.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:43.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:43.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:44.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:44.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:45.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:45.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:46.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:46.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:47.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:47.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:48.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:48.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:49.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:49.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:50.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:50.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:51.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:51.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:52.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:52.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:53.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:53.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:54.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:54.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:55.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:55.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:56.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:56.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:57.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:57.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:58.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:58.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:49:59.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:49:59.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:00.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:00.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:01.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:01.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:02.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:02.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:03.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:03.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:04.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:04.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:05.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:05.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:06.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:06.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:07.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:07.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:08.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:08.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:09.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:09.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:10.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:10.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:11.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:11.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:12.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:12.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:13.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:13.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:14.425 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:50:14.425 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:50:14.425 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:50:14.426 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:50:14.426 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:50:14.435 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:50:14.435 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:50:14.435 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:14.436 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:14.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:14.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:15.446 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":870,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:15.468 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":870,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:15.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:15.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:16.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:16.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:17.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:17.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:18.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:18.469 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:18.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:18.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:19.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:19.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:20.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:20.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:21.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:21.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:22.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:22.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:23.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:23.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:24.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:24.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:25.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:25.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:26.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:26.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:27.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:27.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:28.047 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:28.079 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:28.079 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:50:28.080 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:28.080 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:50:28.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:28.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:29.079 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":880,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:50:29.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:29.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:50:29.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:29.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:30.117 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":880,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:50:30.139 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:30.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:50:30.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:30.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:31.117 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":880,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:50:31.147 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:50:31.148 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:31.148 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:50:31.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:31.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:32.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:50:32.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:32.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:33.265 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:50:33.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:33.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:33.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:33.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:34.242 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:34.272 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:34.272 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:50:34.273 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:50:34.273 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:34.273 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:50:34.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:34.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:35.274 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":880,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:50:35.291 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:35.291 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:50:35.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:35.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:36.291 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":880,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:50:36.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:36.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:50:36.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:36.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:37.309 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":880,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:50:37.331 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:50:37.331 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:50:37.331 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:50:37.332 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:50:37.332 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:50:37.332 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:50:37.351 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:50:37.351 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:50:37.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:37.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:37.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:37.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:38.311 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:38.341 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:38.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:38.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:39.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:39.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:40.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:40.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:41.342 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:41.342 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:41.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:41.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:42.333 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:42.356 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:42.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:42.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:43.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:43.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:44.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:44.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:45.357 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:45.357 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:45.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:45.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:46.347 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:46.369 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":880,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:46.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:46.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:47.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:47.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:48.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:48.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:49.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:49.370 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:50:49.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:49.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:50.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:50.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:51.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:51.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:52.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:52.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:53.433 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":882,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:53.456 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":882,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:50:53.456 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:50:53.456 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:53.456 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:50:53.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:53.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:54.475 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":882,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:50:54.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:50:54.508 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:50:55.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:55.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:55.489 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":882,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:50:55.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:50:56.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:56.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:57.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:57.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:58.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:58.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:59.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:59.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:50:59.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:50:59.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:00.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:00.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:01.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:01.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:02.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:02.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:03.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:03.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:04.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:04.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:05.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:05.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:07.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:07.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:07.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:07.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:09.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:09.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:10.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:10.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:11.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:11.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:12.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:12.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:13.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:13.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:14.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:14.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:14.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:14.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:15.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:15.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:17.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:17.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:18.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:18.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:19.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:19.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:20.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:20.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:21.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:21.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:22.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:22.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:23.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:23.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:24.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:24.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:25.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:25.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:26.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:26.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:27.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:27.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:28.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:28.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:29.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:29.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:30.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:30.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:31.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:31.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:32.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:32.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:33.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:33.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:34.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:34.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:35.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:35.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:36.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:36.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:37.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:37.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:38.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:38.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:39.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:39.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:40.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:40.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:41.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:41.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:42.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:42.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:43.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:43.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:44.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:44.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:45.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:45.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:46.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:46.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:47.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:47.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:48.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:48.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:49.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:49.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:50.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:50.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:51.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:51.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:52.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:52.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:53.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:53.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:54.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:54.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:55.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:55.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:56.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:56.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:57.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:57.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:58.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:58.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:51:59.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:51:59.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:00.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:00.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:01.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:01.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:02.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:02.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:03.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:03.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:04.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:04.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:05.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:05.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:06.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:06.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:07.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:07.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:08.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:08.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:09.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:09.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:10.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:10.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:11.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:11.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:12.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:12.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:13.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:13.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:14.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:14.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:15.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:15.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:16.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:16.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:17.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:17.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:18.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:18.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:19.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:19.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:20.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:20.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:21.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:21.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:22.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:22.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:23.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:23.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:24.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:24.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:25.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:25.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:25.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:52:25.513 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:52:25.513 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:52:25.513 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:52:25.514 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:52:25.520 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:52:25.520 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:52:25.520 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:25.521 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:52:26.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:26.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:26.530 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":882,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:26.552 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":882,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:27.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:27.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:28.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:28.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:29.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:29.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:29.552 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:29.553 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:52:30.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:30.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:31.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:31.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:32.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:32.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:33.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:33.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:34.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:34.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:35.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:35.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:36.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:36.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:37.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:37.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:38.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:38.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:39.131 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:39.153 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:39.153 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:52:39.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:39.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:52:39.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:39.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:40.035 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":892,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:52:40.067 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:40.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:52:40.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:40.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:41.082 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":892,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:52:41.111 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:41.112 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:52:41.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:41.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:42.038 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":892,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:52:42.061 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:52:42.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:42.061 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:52:42.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:42.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:43.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:52:43.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:43.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:44.129 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:52:44.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:44.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:52:44.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:44.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:45.127 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:45.157 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:52:45.157 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:52:45.157 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:52:45.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:45.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:52:45.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:45.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:46.165 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":892,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:52:46.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:46.184 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:52:46.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:46.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:47.145 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":892,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:52:47.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:52:47.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:52:47.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:47.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:48.167 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":892,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:52:48.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T05:52:48.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T05:52:48.189 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T05:52:48.189 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:52:48.190 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T05:52:48.190 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:52:48.209 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T05:52:48.209 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:52:48.210 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:52:48.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:52:48.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:48.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:49.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:49.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:50.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:50.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:51.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:51.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:52.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:52.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:53.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:53.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:54.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:54.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:55.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:55.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:56.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:56.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:57.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:57.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:58.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:58.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:52:59.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:52:59.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:00.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:00.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:01.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:01.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:02.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:02.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:03.211 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:53:03.240 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:53:03.240 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:53:03.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:53:03.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:53:03.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:03.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:04.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:04.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:05.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:05.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:06.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:06.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:07.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:07.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:08.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:08.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:09.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:09.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:10.446 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":894,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:53:10.469 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":894,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:53:10.469 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:53:10.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:53:10.469 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:53:10.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:10.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:11.451 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":894,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:53:11.474 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:53:11.474 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:53:11.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:11.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:12.473 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":894,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:53:12.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:53:12.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:12.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:13.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:13.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:14.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:14.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:15.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:15.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:16.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:16.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:17.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:17.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:18.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:18.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:19.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:19.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:20.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:20.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:21.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:21.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:22.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:22.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:23.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:23.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:24.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:24.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:25.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:25.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:26.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:26.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:27.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:27.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:28.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:28.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:29.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:29.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:30.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:30.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:31.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:31.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:32.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:32.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:33.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:33.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:34.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:34.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:35.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:35.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:36.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:36.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:37.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:37.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:38.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:38.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:39.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:39.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:40.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:40.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:41.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:41.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:42.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:42.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:43.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:43.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:44.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:44.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:45.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:45.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:46.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:46.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:47.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:47.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:48.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:48.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:49.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:49.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:50.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:50.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:51.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:51.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:52.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:52.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:53.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:53.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:54.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:54.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:55.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:55.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:56.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:56.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:57.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:57.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:58.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:58.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:53:59.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:53:59.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:00.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:00.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:01.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:01.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:02.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:02.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:03.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:03.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:04.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:04.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:05.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:05.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:06.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:06.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:07.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:07.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:08.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:08.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:09.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:09.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:10.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:10.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:11.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:11.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:12.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:12.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:13.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:13.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:14.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:14.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:16.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:15.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:17.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:17.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:18.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:18.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:19.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:19.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:20.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:20.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:21.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:21.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:22.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:22.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:23.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:23.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:24.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:24.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:25.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:25.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:26.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:26.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:27.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:27.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:28.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:28.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:29.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:29.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:30.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:30.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:31.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:31.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:32.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:32.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:33.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:33.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:34.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:34.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:35.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:35.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:36.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:36.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:37.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:37.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:38.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:38.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:39.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:39.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:40.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:40.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:41.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:41.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:42.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:42.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:42.506 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T05:54:42.506 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T05:54:42.506 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T05:54:42.506 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T05:54:42.506 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T05:54:42.512 INFO:tasks.ceph.mon.b:Started 2019-10-01T05:54:42.512 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T05:54:42.513 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T05:54:42.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T05:54:43.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:43.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:44.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:44.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:45.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:45.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:46.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:46.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:47.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:47.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:48.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:48.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:49.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:49.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:50.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:50.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:51.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:51.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:52.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:52.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:53.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:53.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:54.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:54.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:54.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:54.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:55.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:55.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:57.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:56.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:57.513 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T05:54:57.514 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T05:54:57.514 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:54:57.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:54:57.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:54:57.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:57.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:58.584 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:54:58.611 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:54:58.611 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:54:58.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:54:58.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:54:58.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:58.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:54:59.586 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":900,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:54:59.608 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:54:59.608 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:54:59.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:54:59.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:00.584 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":900,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:55:00.607 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:00.607 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:55:00.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:00.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:01.553 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":900,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:55:01.576 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:55:01.576 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:01.576 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:55:01.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:01.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:02.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:55:02.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:02.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:03.696 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:55:03.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:03.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:55:03.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:03.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:04.698 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:04.723 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:04.723 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:55:04.723 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:55:04.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:04.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:55:04.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:04.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:05.798 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":900,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:55:05.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:05.817 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:55:05.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:05.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:06.771 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":900,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:55:06.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:06.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:55:06.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:06.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:07.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":900,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:55:07.789 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:55:07.789 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:55:07.789 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:55:07.789 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:55:07.789 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:55:07.790 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:55:07.810 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:55:07.810 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:55:07.811 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:07.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:55:07.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:07.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:08.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:08.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:08.871 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:08.893 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:09.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:09.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:10.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:10.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:11.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:11.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:11.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:11.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:55:12.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:12.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:12.882 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:12.903 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:13.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:13.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:14.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:14.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:15.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:15.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:15.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:15.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:55:16.867 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:16.890 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":900,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:16.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:16.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:17.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:17.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:18.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:18.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:19.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:19.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:19.891 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:19.891 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:55:20.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:20.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:21.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:21.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:22.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:22.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:23.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:23.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:23.925 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":904,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:23.948 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":904,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:55:23.948 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:55:23.948 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:23.948 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:55:24.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:24.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:24.965 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":904,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:55:24.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:55:24.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:55:25.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:25.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:26.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":904,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:55:26.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:55:26.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:26.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:27.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:27.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:28.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:28.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:29.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:29.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 55 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:30.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:30.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:31.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:31.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:32.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:32.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:33.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:33.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:34.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:34.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:35.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:35.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:36.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:36.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:37.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:37.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:38.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:38.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:39.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:39.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:40.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:40.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:41.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:41.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:42.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:42.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:43.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:43.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:44.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:44.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:45.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:45.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:46.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:46.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:47.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:47.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:48.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:48.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:49.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:49.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:50.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:50.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:51.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:51.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:52.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:52.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:53.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:53.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:54.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:54.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:55.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:55.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:56.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:56.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:57.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:57.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:58.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:58.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:55:59.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:55:59.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:00.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:00.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:01.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:01.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:02.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:02.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:03.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:03.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:04.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:04.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:05.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:05.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:06.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:06.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:07.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:07.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:08.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:08.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:09.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:09.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:10.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:10.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:11.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:11.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:12.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:12.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:13.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:13.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:14.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:14.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:15.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:15.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:16.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:16.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:17.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:17.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:18.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:18.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:19.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:19.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:20.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:20.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:21.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:21.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:22.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:22.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:23.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:23.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:24.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:24.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:25.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:25.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:26.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:26.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:27.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:27.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:28.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:28.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:29.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:29.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:30.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:30.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:31.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:31.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:32.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:32.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:33.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:33.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:34.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:34.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:35.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:35.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:36.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:36.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:37.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:37.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:38.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:38.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:39.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:39.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:40.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:40.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:41.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:41.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:42.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:42.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:43.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:43.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:44.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:44.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:45.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:45.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:46.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:46.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:47.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:47.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:48.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:48.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:49.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:49.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:50.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:50.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:51.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:51.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:52.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:52.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:53.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:53.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:54.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:54.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:55.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:55.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:56.042 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:56:56.042 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:56:56.042 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:56:56.043 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:56:56.043 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:56:56.053 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:56:56.053 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:56:56.053 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:56:56.053 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:56:56.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:56.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:57.053 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":904,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:56:57.081 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":904,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:56:57.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:57.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:58.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:58.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:56:59.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:56:59.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:00.082 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:00.083 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:00.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:00.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:01.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:01.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:02.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:02.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:03.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:03.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:03.596 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T05:57:03.584+0000 7fada8a19700 0 --2- 172.21.6.124:0/3995576790 >> [v2:172.21.6.124:3300/0,v1:172.21.6.124:6789/0] conn(0x7fad8c0092d0 0x7fad8c00b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T05:57:04.224 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:04.245 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:04.245 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:57:04.245 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:04.245 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:57:04.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:04.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:05.233 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":912,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:57:05.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:05.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:57:05.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:05.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:06.268 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":912,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:57:06.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:06.290 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:57:06.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:06.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:07.273 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":912,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:57:07.296 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:57:07.296 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:07.296 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:57:07.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:07.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:08.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:57:08.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:08.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:09.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:09.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:09.420 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:57:09.420 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:09.420 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:10.401 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:10.418 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:10.418 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:57:10.418 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:57:10.418 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:10.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:57:10.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:10.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:11.444 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":912,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:57:11.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:11.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:57:11.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:11.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:12.456 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":912,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:57:12.486 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:12.486 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:57:12.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:12.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:13.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:13.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:13.551 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":912,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:57:13.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T05:57:13.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:57:13.576 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T05:57:13.577 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:57:13.577 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T05:57:13.577 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:57:13.632 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T05:57:13.632 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:57:13.632 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:13.632 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:14.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:14.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:14.716 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:14.741 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:15.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:15.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:16.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:16.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:17.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:17.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:17.742 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:17.742 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:18.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:18.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:18.733 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:18.755 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:19.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:19.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:20.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:20.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:21.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:21.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:21.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:21.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:22.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:22.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:22.763 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:22.786 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":912,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:23.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:23.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:24.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:24.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:25.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:25.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:25.787 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:25.787 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:57:26.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:26.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:27.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:27.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:28.186 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T05:57:28.174+0000 7faaa259c700 0 --2- 172.21.6.124:0/2726242909 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7faa900046d0 0x7faa90004b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T05:57:28.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:28.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:28.823 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":916,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:28.854 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":916,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:57:28.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:57:28.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:28.855 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:57:29.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:29.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:29.865 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":916,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:57:29.897 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:57:29.897 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:57:30.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:30.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:30.870 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":916,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:57:30.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:57:31.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:31.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:32.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:32.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:33.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:33.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:34.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:34.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:35.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:35.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:36.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:36.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:37.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:37.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:38.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:38.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:39.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:39.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:40.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:40.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:41.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:41.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:42.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:42.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:43.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:43.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:44.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:44.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:45.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:45.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:46.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:46.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:47.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:47.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:48.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:48.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:49.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:49.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:50.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:50.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:51.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:51.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:52.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:52.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:53.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:53.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:54.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:54.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:55.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:55.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:56.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:56.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:57.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:57.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:58.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:58.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:57:59.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:57:59.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:00.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:00.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:01.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:01.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:02.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:02.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:03.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:03.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:04.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:04.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:05.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:05.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:06.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:06.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:07.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:07.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:08.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:08.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:09.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:09.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:10.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:10.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:11.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:11.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:12.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:12.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:13.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:13.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:14.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:14.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:15.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:15.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:16.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:16.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:17.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:17.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:18.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:18.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:19.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:19.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:20.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:20.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:21.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:21.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:22.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:22.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:23.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:23.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:24.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:24.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:25.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:25.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:26.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:26.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:27.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:27.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:28.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:28.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:29.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:29.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:30.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:30.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:31.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:31.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:32.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:32.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:33.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:33.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:34.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:34.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:35.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:35.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:36.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:36.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:37.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:37.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:38.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:38.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:39.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:39.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:40.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:40.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:41.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:41.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:42.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:42.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:43.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:43.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:44.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:44.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:45.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:45.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:46.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:46.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:47.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:47.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:48.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:48.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:49.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:49.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:50.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:50.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:51.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:51.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:52.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:52.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:53.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:53.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:54.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:54.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:55.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:55.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:56.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:56.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:57.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:57.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:58.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:58.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:58:59.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:58:59.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:00.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:00.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:00.893 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T05:59:00.893 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T05:59:00.893 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T05:59:00.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:00.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T05:59:00.931 INFO:tasks.ceph.mon.a:Started 2019-10-01T05:59:00.931 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:59:00.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:00.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:01.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:01.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:01.942 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":916,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:01.972 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":916,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:02.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:02.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:03.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:03.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:04.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:04.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:04.973 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:04.973 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:05.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:05.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:06.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:06.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:07.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:07.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:08.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:08.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:09.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:09.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:10.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:10.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:10.400 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:10.428 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:10.429 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:59:10.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:10.429 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:59:11.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:11.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:11.446 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":924,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:11.468 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:11.468 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:59:12.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:12.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:12.474 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":924,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:59:12.506 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:12.506 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:59:13.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:13.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:13.505 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":924,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:13.535 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T05:59:13.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:13.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T05:59:14.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:14.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:14.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T05:59:15.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:15.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:15.622 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T05:59:15.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:15.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:16.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:16.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:16.604 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:16.633 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:16.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T05:59:16.633 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T05:59:16.634 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:16.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:59:17.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:17.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:17.631 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":924,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:17.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:17.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T05:59:18.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:18.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:18.712 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":924,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T05:59:18.739 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:18.739 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:59:19.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:19.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:19.721 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":924,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:19.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T05:59:19.743 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T05:59:19.743 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T05:59:19.744 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T05:59:19.744 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T05:59:19.744 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T05:59:19.766 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T05:59:19.767 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T05:59:19.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:19.767 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:20.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:20.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:20.824 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:20.847 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:21.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:21.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:22.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:22.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:23.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:23.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:23.848 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:23.848 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:24.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:24.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:24.844 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:24.874 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":924,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:25.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:25.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:26.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:26.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:27.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:27.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:27.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:27.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T05:59:28.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:28.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:29.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:29.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:30.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:30.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:31.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:31.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:32.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:32.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:33.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:33.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:34.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:34.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:34.747 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":928,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:34.771 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":928,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T05:59:34.771 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T05:59:34.771 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:34.772 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T05:59:35.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:35.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:35.785 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":928,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:35.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T05:59:35.818 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T05:59:36.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:36.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:36.813 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":928,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T05:59:36.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T05:59:37.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:37.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:38.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:38.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:39.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:39.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:40.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:40.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:41.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:41.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:42.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:42.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:43.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:43.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:44.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:44.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:45.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:45.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:46.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:46.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:47.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:47.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:48.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:48.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:49.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:49.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:50.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:50.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:51.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:51.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:52.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:52.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:53.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:53.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:54.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:54.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:55.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:55.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:56.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:56.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:57.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:57.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:58.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:58.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T05:59:59.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T05:59:59.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:00.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:00.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:01.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:01.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:02.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:02.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:03.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:03.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:04.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:04.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:05.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:05.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:06.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:06.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:07.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:07.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:08.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:08.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:09.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:09.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:10.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:10.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:11.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:11.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:12.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:12.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:13.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:13.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:14.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:14.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:15.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:15.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:16.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:16.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:17.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:17.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:18.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:18.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:19.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:19.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:20.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:20.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:21.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:21.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:22.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:22.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:23.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:23.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:24.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:24.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:25.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:25.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:26.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:26.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:26.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:26.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:28.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:28.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:29.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:29.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:30.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:30.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:31.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:31.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:32.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:32.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:33.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:33.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:33.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:33.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:34.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:34.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:36.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:36.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:36.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:36.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:37.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:37.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:38.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:38.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:39.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:39.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:40.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:40.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:42.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:42.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:43.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:43.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:44.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:44.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:45.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:45.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:46.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:46.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:47.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:47.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:47.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:47.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:49.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:49.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:50.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:50.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:51.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:51.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:52.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:52.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:53.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:53.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:54.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:54.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:55.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:55.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:56.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:56.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:57.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:57.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:58.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:58.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:00:59.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:00:59.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:00.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:00.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:01.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:01.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:02.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:02.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:03.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:03.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:04.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:04.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:05.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:05.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:05.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:05.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:06.830 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:01:06.830 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:01:06.830 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:01:06.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:06.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:01:06.838 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:01:06.839 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:01:06.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:06.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:01:06.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:06.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:07.853 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":928,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:07.885 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":928,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:08.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:08.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:08.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:08.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:10.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:10.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:10.886 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:10.887 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:01:10.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:10.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:12.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:12.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:12.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:12.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:13.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:13.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:14.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:14.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:15.782 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:15.812 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:15.813 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:01:15.813 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:15.813 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:01:15.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:15.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:16.810 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":934,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:16.842 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:16.842 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:01:16.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:16.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:17.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:17.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:18.076 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":934,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:18.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:18.099 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:01:18.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:18.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:19.077 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":934,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:19.099 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:01:19.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:19.099 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:01:19.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:19.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:20.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:01:20.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:20.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:21.211 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:01:21.211 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:21.211 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:01:21.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:21.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:22.211 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:22.233 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:22.234 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:01:22.234 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:01:22.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:22.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:01:22.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:22.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:23.290 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":934,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:23.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:23.312 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:01:23.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:23.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:24.289 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":934,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:24.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:24.312 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:01:24.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:24.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:25.281 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":934,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:25.303 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:01:25.303 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T06:01:25.303 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:01:25.303 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:01:25.304 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:01:25.304 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:01:25.322 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:01:25.322 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:01:25.323 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:01:25.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:01:25.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:25.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:26.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:26.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:27.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:27.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:28.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:28.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:30.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:30.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:30.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:30.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:32.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:32.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:33.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:33.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:34.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:34.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:35.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:35.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:35.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:35.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:37.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:37.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:38.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:38.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:38.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:38.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:39.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:39.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:40.324 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:01:40.325 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:01:40.325 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:01:40.325 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:40.325 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:01:41.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:41.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:41.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:41.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:43.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:43.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:44.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:44.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:45.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:45.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:45.970 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":938,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:45.993 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":938,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:01:45.993 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:01:45.993 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:45.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:01:46.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:46.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:47.001 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":938,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:47.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:47.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:47.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:01:47.038 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:01:47.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:47.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:48.117 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":938,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:01:48.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:01:48.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:48.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:49.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:49.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:50.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:50.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:51.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:51.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:52.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:52.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:53.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:53.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:54.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:54.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:55.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:55.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:56.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:56.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:57.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:57.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:58.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:58.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:01:59.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:01:59.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:00.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:00.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:02.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:02.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:02.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:02.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:03.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:03.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:04.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:04.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:06.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:06.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:07.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:07.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:08.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:08.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:09.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:09.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:10.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:10.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:11.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:11.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:12.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:12.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:13.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:13.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:14.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:14.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:15.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:15.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:16.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:16.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:17.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:17.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:18.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:18.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:19.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:19.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:20.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:20.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:21.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:21.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:22.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:22.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:23.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:23.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:24.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:24.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:25.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:25.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:26.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:26.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:27.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:27.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:28.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:28.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:29.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:29.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:30.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:30.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:30.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:30.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:32.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:32.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:33.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:33.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:34.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:34.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:35.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:35.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:36.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:36.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:37.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:37.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:38.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:38.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:39.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:39.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:40.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:40.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:41.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:41.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:42.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:42.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:43.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:43.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:44.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:44.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:45.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:45.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:46.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:46.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:47.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:47.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:48.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:48.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:49.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:49.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:50.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:50.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:51.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:51.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:52.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:52.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:53.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:53.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:54.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:54.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:55.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:55.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:56.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:56.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:57.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:57.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:58.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:58.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:02:59.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:02:59.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:00.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:00.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:00.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:00.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:02.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:02.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:03.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:03.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:04.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:04.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:05.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:05.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:06.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:06.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:07.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:07.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:08.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:08.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:09.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:09.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:10.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:10.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:11.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:11.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:12.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:12.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:13.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:13.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:14.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:14.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:15.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:15.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:16.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:16.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:17.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:17.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:18.148 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:03:18.149 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:03:18.149 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:03:18.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:18.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:03:18.155 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:03:18.155 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:03:18.155 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:03:18.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:03:18.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:18.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:19.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:19.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:20.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:20.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:21.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:21.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:22.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:22.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:23.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:23.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:24.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:24.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:25.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:25.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:26.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:26.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:27.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:27.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:28.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:28.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:29.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:29.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:30.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:30.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:31.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:31.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:32.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:32.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:33.156 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:03:33.156 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:03:33.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:03:33.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:33.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:03:33.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:33.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:34.177 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:34.196 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:34.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:03:34.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:34.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:03:34.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:34.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:35.166 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":942,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:35.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:35.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:03:35.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:35.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:36.217 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":942,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:36.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:36.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:36.248 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:36.248 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:03:37.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:37.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:37.293 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":942,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:37.315 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:03:37.315 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:37.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:03:38.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:38.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:38.411 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:03:39.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:39.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:39.413 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:03:39.413 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:39.413 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:03:40.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:40.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:40.411 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:40.444 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:40.445 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:03:40.445 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:03:40.445 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:40.445 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:03:41.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:41.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:41.464 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":942,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:41.495 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:41.495 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:03:42.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:42.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:42.491 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":942,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:42.526 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:42.526 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:03:43.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:43.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:43.518 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":942,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:03:43.541 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:03:43.541 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:03:43.541 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:03:43.541 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:03:43.541 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:03:43.542 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:03:43.573 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:03:43.573 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:03:43.573 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:43.574 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:03:44.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:44.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:44.561 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:44.583 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:45.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:45.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:46.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:46.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:47.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:47.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:47.584 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:47.584 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:03:48.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:48.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:48.543 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:48.574 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":942,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:49.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:49.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:50.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:50.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:51.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:51.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:51.574 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:51.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:03:52.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:52.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:53.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:53.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:54.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:54.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:55.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:55.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:56.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:56.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:57.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:57.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:58.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:58.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:03:58.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:59.012 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:03:59.012 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:03:59.012 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:03:59.012 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:03:59.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:03:59.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:00.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":944,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:04:00.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:04:00.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:04:00.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:00.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:01.035 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":944,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:04:01.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:04:01.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:01.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:02.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:02.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:03.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:03.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:04.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:04.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:05.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:05.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:06.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:06.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:07.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:07.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:08.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:08.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:09.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:09.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:10.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:10.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:11.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:11.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:12.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:12.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:13.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:13.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:14.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:14.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:15.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:15.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:16.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:16.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:17.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:17.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:18.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:18.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:19.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:19.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:20.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:20.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:21.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:21.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:22.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:22.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:23.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:23.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:24.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:24.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:25.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:25.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:26.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:26.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:27.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:27.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:28.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:28.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:29.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:29.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:30.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:30.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:31.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:31.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:32.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:32.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:33.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:33.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:34.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:34.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:35.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:35.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:36.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:36.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:37.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:37.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:38.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:38.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:39.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:39.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:40.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:40.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:40.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:40.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:42.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:42.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:43.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:43.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:43.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:43.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:44.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:44.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:45.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:45.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:47.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:47.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:48.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:48.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:49.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:49.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:50.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:50.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:51.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:51.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:52.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:52.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:53.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:53.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:54.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:54.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:54.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:54.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:55.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:55.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:56.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:56.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:57.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:57.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:58.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:58.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:04:59.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:04:59.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:00.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:00.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:01.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:01.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:02.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:02.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:03.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:03.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:04.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:04.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:05.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:05.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:06.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:06.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:07.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:07.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:08.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:08.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:09.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:09.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:10.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:10.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:11.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:11.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:12.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:12.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:13.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:13.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:14.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:14.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:15.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:15.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:16.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:16.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:17.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:17.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:18.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:18.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:19.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:19.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:20.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:20.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:21.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:21.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:22.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:22.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:24.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:24.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:25.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:25.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:26.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:26.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:27.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:27.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:28.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:28.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:29.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:29.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:30.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:30.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:31.058 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:05:31.058 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:05:31.058 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:05:31.058 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:05:31.058 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:05:31.064 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:05:31.064 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:05:31.064 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:31.064 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:31.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:31.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:32.044 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:32.075 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:32.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:32.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:33.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:33.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:34.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:34.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:35.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:35.076 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:35.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:35.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:36.075 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:36.097 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:36.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:36.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:37.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:37.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:38.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:38.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:39.098 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:39.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:39.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:39.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:40.104 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:40.126 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:40.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:40.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:41.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:41.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:42.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:42.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:43.127 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:43.127 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:43.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:43.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:44.121 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:44.148 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":944,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:44.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:44.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:45.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:45.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:46.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:46.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:47.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:47.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:47.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:47.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:48.110 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:48.142 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:48.142 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:05:48.143 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:48.143 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:05:48.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:48.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:49.127 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":948,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:05:49.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:49.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:05:49.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:49.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:50.164 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":948,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:05:50.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:50.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:05:50.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:50.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:51.197 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":948,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:05:51.218 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:05:51.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:51.220 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:05:51.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:51.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:52.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:05:52.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:52.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:53.337 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:05:53.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:53.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:53.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:53.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:54.331 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:54.355 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:54.355 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:05:54.355 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:05:54.356 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:54.356 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:05:54.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:54.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:55.372 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":948,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:05:55.403 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:55.403 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:05:55.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:55.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:56.386 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":948,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:05:56.409 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:56.409 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:05:56.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:56.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:57.332 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":948,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:05:57.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:05:57.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:05:57.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:05:57.353 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:05:57.353 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:05:57.353 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:05:57.373 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:05:57.373 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:05:57.373 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:05:57.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:05:57.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:57.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:58.418 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:58.440 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:05:58.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:58.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:05:59.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:05:59.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:00.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:00.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:01.441 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:06:01.441 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:06:01.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:01.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:02.426 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:02.460 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:02.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:02.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:03.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:03.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:04.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:04.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:05.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:05.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:05.461 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:06:05.461 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:06:06.460 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:06.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:06.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:06.492 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":948,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:07.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:07.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:08.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:08.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:09.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:06:09.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:06:09.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:09.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:10.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:10.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:11.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:11.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:12.469 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":952,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:12.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:12.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:12.491 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":952,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:06:12.492 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:06:12.492 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:06:12.492 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:06:13.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:13.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:13.477 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":952,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:06:13.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:06:13.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:06:14.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:14.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:14.485 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":952,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:06:14.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:06:15.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:15.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:16.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:16.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:17.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:17.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:18.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:18.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:19.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:19.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:20.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:20.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:21.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:21.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:22.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:22.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:23.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:23.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:24.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:24.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:25.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:25.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:26.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:26.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:27.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:27.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:28.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:28.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:29.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:29.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:30.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:30.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:31.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:31.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:32.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:32.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:33.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:33.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:34.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:34.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:35.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:35.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:36.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:36.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:37.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:37.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:38.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:38.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:39.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:39.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:40.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:40.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:41.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:41.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:42.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:42.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:43.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:43.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:44.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:44.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:45.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:45.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:46.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:46.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:47.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:47.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:48.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:48.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:49.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:49.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:50.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:50.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:51.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:51.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:52.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:52.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:53.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:53.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:54.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:54.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:55.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:55.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:56.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:56.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:57.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:57.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:58.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:58.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:06:59.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:06:59.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:00.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:00.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:01.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:01.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:02.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:02.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:03.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:03.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:04.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:04.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:05.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:05.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:06.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:06.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:07.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:07.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:08.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:08.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:09.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:09.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:10.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:10.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:11.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:11.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:12.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:12.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:13.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:13.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:14.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:14.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:15.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:15.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:16.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:16.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:17.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:17.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:18.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:18.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:19.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:19.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:20.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:20.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:21.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:21.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:22.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:22.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:23.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:23.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:24.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:24.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:25.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:25.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:26.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:26.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:27.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:27.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:28.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:28.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:29.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:29.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:30.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:30.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:31.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:31.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:32.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:32.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:33.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:33.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:34.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:34.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:35.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:35.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:36.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:36.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:37.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:37.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:38.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:38.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:39.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:39.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:40.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:40.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:41.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:41.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:42.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:42.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:43.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:43.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:44.515 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:07:44.515 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:07:44.515 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:07:44.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:44.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:07:44.521 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:07:44.521 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:07:44.521 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:44.521 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:07:44.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:44.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:45.510 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":952,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:07:45.532 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":952,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:07:45.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:45.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:46.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:46.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:47.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:47.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:48.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:48.533 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:07:48.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:48.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:49.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:49.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:50.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:50.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:51.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:51.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:52.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:52.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:53.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:53.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:54.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:54.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:55.732 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:07:55.764 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:07:55.765 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:07:55.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:55.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:07:55.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:55.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:56.783 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":958,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:07:56.813 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:56.813 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:07:56.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:56.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:57.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:57.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:57.891 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":958,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:07:57.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:57.914 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:07:58.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:58.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:07:58.930 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":958,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:07:58.957 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:07:58.957 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:07:58.957 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:07:59.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:07:59.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:00.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:08:00.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:00.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:01.082 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:08:01.082 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:01.082 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:08:01.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:01.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:02.065 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:02.087 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:02.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:08:02.088 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:08:02.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:02.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:08:02.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:02.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:03.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":958,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:08:03.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:03.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:08:03.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:03.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:04.092 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":958,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:08:04.113 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:04.113 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:08:04.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:04.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:05.050 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":958,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:08:05.073 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:08:05.073 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:08:05.073 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:08:05.073 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:08:05.074 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:08:05.074 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:08:05.106 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:08:05.106 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:08:05.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:05.106 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:08:05.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:05.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:06.162 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:06.184 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:06.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:06.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:07.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:07.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:08.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:08.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:09.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:09.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:08:09.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:09.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:10.186 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:10.217 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:10.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:10.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:11.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:11.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:12.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:12.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:13.218 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:13.218 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:08:13.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:13.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:14.232 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:14.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":958,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:14.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:14.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:15.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:15.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:16.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:16.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:17.260 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:17.260 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:08:17.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:17.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:18.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:18.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:19.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:19.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:20.161 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:20.191 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:08:20.191 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:08:20.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:20.192 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:08:20.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:20.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:21.132 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":962,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:08:21.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:08:21.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:08:21.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:21.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:22.068 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":962,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:08:22.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:08:22.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:22.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:23.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:23.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:24.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:24.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:25.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:25.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:26.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:26.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:27.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:27.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:28.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:28.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:29.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:29.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:30.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:30.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:31.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:31.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:32.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:32.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:33.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:33.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:34.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:34.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:35.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:35.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:36.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:36.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:37.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:37.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:38.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:38.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:39.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:39.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:40.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:40.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:41.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:41.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:42.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:42.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:43.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:43.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:44.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:44.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:45.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:45.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:46.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:46.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:47.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:47.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:48.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:48.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:49.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:49.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:50.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:50.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:51.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:51.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:52.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:52.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:53.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:53.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:54.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:54.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:55.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:55.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:56.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:56.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:57.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:57.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:58.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:58.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:08:59.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:08:59.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:00.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:00.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:01.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:01.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:02.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:02.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:03.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:03.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:04.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:04.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:05.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:05.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:06.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:06.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:07.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:07.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:08.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:08.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:09.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:09.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:10.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:10.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:11.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:11.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:12.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:12.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:13.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:13.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:14.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:14.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:15.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:15.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:16.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:16.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:17.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:17.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:18.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:18.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:19.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:19.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:20.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:20.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:21.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:21.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:22.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:22.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:23.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:23.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:24.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:24.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:25.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:25.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:26.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:26.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:27.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:27.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:28.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:28.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:29.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:29.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:30.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:30.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:31.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:31.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:32.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:32.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:33.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:33.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:34.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:34.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:35.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:35.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:36.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:36.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:37.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:37.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:38.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:38.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:39.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:39.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:40.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:40.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:41.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:41.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:42.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:42.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:43.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:43.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:44.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:44.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:45.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:45.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:46.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:46.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:47.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:47.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:48.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:48.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:49.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:49.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:50.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:50.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:51.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:51.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:52.091 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:09:52.091 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:09:52.091 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:09:52.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:09:52.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:09:52.097 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:09:52.097 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:09:52.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:09:52.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:09:52.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:52.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:53.053 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:09:53.085 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:09:53.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:53.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:54.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:54.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:55.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:55.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:56.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:09:56.086 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:09:56.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:56.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:57.098 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:09:57.126 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:09:57.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:57.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:58.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:58.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:09:59.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:09:59.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:00.127 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:00.127 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:10:00.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:00.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:01.163 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:01.193 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":962,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:01.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:01.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:02.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:02.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:03.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:03.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:04.194 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:04.194 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:10:04.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:04.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:05.223 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":964,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:05.255 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":964,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:05.255 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:10:05.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:05.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:10:05.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:05.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:06.244 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":964,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:06.268 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:06.268 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:10:06.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:06.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:07.245 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":964,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:07.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:07.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:10:07.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:07.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:08.249 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":964,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:08.271 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:10:08.272 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:08.272 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:10:08.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:08.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:09.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:10:09.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:09.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:10.408 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:10:10.409 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:10.409 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:10:10.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:10.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:11.426 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":964,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:11.448 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":964,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:11.448 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:10:11.448 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:10:11.449 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:11.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:10:11.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:11.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:12.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:12.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:12.440 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":964,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:12.468 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:12.468 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:10:13.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:13.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:13.469 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":964,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:13.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:13.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:10:14.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:14.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:14.453 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":964,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:14.475 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:10:14.476 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T06:10:14.476 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:10:14.476 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:10:14.476 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:10:14.476 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:10:14.501 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:10:14.501 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:10:14.502 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:10:14.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:10:15.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:15.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:16.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:16.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:17.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:17.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:18.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:18.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:19.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:19.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:20.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:20.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:21.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:21.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:22.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:22.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:23.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:23.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:24.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:24.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:25.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:25.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:26.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:26.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:27.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:27.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:28.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:28.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:29.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:29.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:29.502 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:10:29.502 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:10:29.503 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:10:29.503 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:29.503 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:10:30.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:30.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:31.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:31.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:32.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:32.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:33.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:33.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:34.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:34.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:35.165 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":966,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:35.187 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":966,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:10:35.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:10:35.188 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:35.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:10:35.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:35.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:36.150 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":966,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:36.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:10:36.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:10:36.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:36.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:37.162 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":966,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:10:37.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:10:37.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:37.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:38.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:38.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:39.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:39.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:40.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:40.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:41.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:41.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:42.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:42.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:43.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:43.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:44.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:44.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:45.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:45.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:46.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:46.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:47.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:47.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:48.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:48.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:49.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:49.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:50.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:50.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:51.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:51.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:52.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:52.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:53.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:53.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:54.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:54.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:55.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:55.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:56.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:56.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:57.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:57.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:58.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:58.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:10:59.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:10:59.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:00.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:00.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:01.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:01.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:02.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:02.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:03.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:03.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:04.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:04.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:05.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:05.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:06.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:06.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:07.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:07.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:08.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:08.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:09.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:09.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:10.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:10.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:11.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:11.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:12.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:12.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:13.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:13.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:14.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:14.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:15.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:15.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:16.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:16.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:17.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:17.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:18.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:18.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:19.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:19.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:20.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:20.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:21.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:21.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:22.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:22.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:23.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:23.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:24.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:24.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:25.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:25.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:26.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:26.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:27.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:27.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:28.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:28.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:29.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:29.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:30.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:30.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:31.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:31.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:32.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:32.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:33.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:33.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:34.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:34.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:35.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:35.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:36.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:36.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:37.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:37.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:38.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:38.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:39.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:39.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:40.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:40.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:41.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:41.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:42.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:42.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:43.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:43.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:44.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:44.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:45.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:45.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:46.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:46.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:47.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:47.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:48.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:48.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:49.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:49.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:50.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:50.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:51.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:51.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:52.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:52.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:53.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:53.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:54.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:54.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:55.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:55.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:56.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:56.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:57.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:57.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:58.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:58.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:11:59.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:11:59.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:00.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:00.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:01.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:01.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:02.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:02.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:03.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:03.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:04.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:04.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:05.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:05.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:06.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:06.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:07.195 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:12:07.195 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:12:07.195 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:12:07.195 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:12:07.196 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:12:07.201 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:12:07.202 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:12:07.202 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:12:07.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:12:07.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:07.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:08.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:08.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:09.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:09.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:10.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:10.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:11.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:11.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:12.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:12.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:13.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:13.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:14.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:14.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:15.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:15.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:16.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:16.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:17.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:17.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:18.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:18.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:19.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:19.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:20.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:20.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:21.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:21.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:22.203 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:12:22.203 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:12:22.204 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:12:22.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:22.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:22.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:22.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:23.272 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:23.295 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:23.295 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:12:23.295 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:23.295 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:12:23.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:23.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:24.284 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":974,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:24.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:24.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:12:24.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:24.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:25.301 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":974,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:25.324 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:25.324 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:12:25.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:25.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:26.329 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":974,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:26.359 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:12:26.360 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:26.360 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:12:26.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:26.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:27.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:27.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:27.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:12:28.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:28.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:28.476 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:12:28.476 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:28.476 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:29.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:29.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:29.469 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:29.499 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:29.500 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:12:29.500 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:12:29.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:29.500 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:12:30.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:30.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:30.463 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":974,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:30.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:30.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:12:31.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:31.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:31.502 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":974,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:31.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:31.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:12:32.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:32.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:32.526 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":974,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:32.553 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:12:32.553 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:12:32.553 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:12:32.553 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:12:32.554 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:12:32.554 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:12:32.569 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:12:32.569 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:12:32.570 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:32.570 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:33.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:33.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:33.640 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:33.663 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:34.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:34.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:35.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:35.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:36.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:36.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:36.664 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:36.664 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:37.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:37.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:37.737 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:37.761 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:38.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:38.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:39.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:39.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:40.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:40.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:40.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:40.762 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:41.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:41.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:41.756 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:41.779 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":974,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:42.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:42.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:43.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:43.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:44.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:44.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:44.779 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:44.780 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:12:45.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:45.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:46.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:46.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:47.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:47.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:48.223 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":978,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:48.245 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":978,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:12:48.245 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:12:48.246 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:48.246 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:12:48.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:48.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:49.244 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":978,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:12:49.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:12:49.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:12:49.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:49.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:50.280 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":978,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:12:50.302 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:12:50.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:50.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:51.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:51.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:52.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:52.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:53.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:53.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:54.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:54.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:55.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:55.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:56.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:56.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:57.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:57.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:58.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:58.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:12:59.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:12:59.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:00.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:00.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:01.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:01.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:02.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:02.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:03.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:03.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:04.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:04.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:05.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:05.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:06.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:06.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:07.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:07.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:08.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:08.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:09.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:09.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:10.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:10.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:11.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:11.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:12.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:12.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:13.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:13.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:14.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:14.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:15.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:15.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:16.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:16.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:17.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:17.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:18.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:18.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:19.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:19.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:20.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:20.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:21.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:21.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:22.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:22.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:23.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:23.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:24.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:24.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:25.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:25.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:26.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:26.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:27.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:27.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:28.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:28.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:29.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:29.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:30.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:30.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:31.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:31.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:32.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:32.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:33.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:33.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:34.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:34.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:35.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:35.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:36.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:36.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:37.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:37.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:38.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:38.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:39.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:39.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:40.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:40.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:41.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:41.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:42.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:42.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:43.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:43.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:44.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:44.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:45.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:45.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:46.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:46.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:47.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:47.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:48.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:48.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:49.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:49.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:50.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:50.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:51.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:51.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:52.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:52.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:53.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:53.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:54.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:54.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:55.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:55.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:56.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:56.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:57.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:57.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:58.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:58.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:13:59.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:13:59.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:00.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:00.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:01.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:01.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:02.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:02.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:03.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:03.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:04.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:04.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:05.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:05.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:06.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:06.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:07.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:07.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:08.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:08.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:09.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:09.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:10.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:10.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:11.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:11.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:12.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:12.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:13.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:13.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:14.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:14.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:15.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:15.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:16.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:16.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:17.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:17.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:18.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:18.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:19.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:19.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:20.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:20.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:20.304 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:14:20.305 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:14:20.305 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:14:20.305 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:20.305 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:14:20.310 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:14:20.311 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:14:20.311 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:20.311 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:14:21.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:21.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:21.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":978,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:21.289 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":978,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:22.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:22.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:23.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:23.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:24.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:24.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:24.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:24.290 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:14:25.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:25.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:26.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:26.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:27.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:27.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:28.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:28.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:29.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:29.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:29.206 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:29.228 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:30.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:30.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:31.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:31.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:32.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:32.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:32.229 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:32.229 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:14:33.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:33.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:33.227 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":984,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:33.259 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":984,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:33.259 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:14:33.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:33.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:14:34.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:34.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:34.269 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":984,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:14:34.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:34.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:14:35.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:35.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:35.268 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":984,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:14:35.294 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:35.294 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:14:36.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:36.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:36.275 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":984,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:14:36.306 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:14:36.306 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:36.306 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:14:36.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:36.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:37.394 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:14:37.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:37.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:38.395 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:14:38.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:38.396 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:14:38.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:38.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:39.408 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":984,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:39.441 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":984,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:14:39.441 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:14:39.442 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:14:39.442 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:39.442 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:14:39.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:39.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:40.441 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":984,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:14:40.458 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:40.458 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:14:40.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:40.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:41.461 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":984,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:14:41.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:41.494 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:14:41.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:41.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:42.474 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":984,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:14:42.495 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:14:42.496 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T06:14:42.496 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:14:42.496 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:14:42.496 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:14:42.496 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:14:42.512 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:14:42.513 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T06:14:42.513 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T06:14:42.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:14:42.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:42.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:43.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:43.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:44.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:44.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:46.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:46.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:47.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:47.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:48.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:48.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:49.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:49.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:50.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:50.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:51.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:51.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:52.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:52.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:53.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:53.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:54.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:54.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:55.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:55.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:56.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:56.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:57.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:57.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:57.514 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T06:14:57.515 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T06:14:57.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:14:57.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:14:57.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:14:58.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:58.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:14:59.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:14:59.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:00.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:00.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:01.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:01.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:02.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:02.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:03.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:03.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:04.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:04.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:04.708 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":986,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:15:04.730 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":986,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:15:04.730 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:15:04.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:15:04.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:15:05.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:05.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:05.735 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":986,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:15:05.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:15:05.768 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:15:06.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:06.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:06.768 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":986,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:15:06.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:15:07.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:07.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:08.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:08.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:09.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:09.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:10.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:10.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:11.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:11.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:12.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:12.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:13.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:13.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:14.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:14.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:15.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:15.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:16.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:16.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:17.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:17.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:18.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:18.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:19.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:19.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:20.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:20.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:21.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:21.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:22.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:22.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:23.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:23.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:24.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:24.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:25.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:25.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:26.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:26.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:27.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:27.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:28.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:28.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:29.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:29.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:30.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:30.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:31.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:31.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:32.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:32.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:33.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:33.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:34.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:34.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:35.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:35.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:36.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:36.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:37.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:37.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:38.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:38.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:39.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:39.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:40.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:40.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:41.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:41.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:42.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:42.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:43.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:43.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:44.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:44.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:45.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:45.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:46.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:46.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:47.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:47.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:48.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:48.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:49.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:49.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:50.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:50.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:51.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:51.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:52.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:52.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:53.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:53.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:54.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:54.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:55.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:55.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:56.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:56.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:57.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:57.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:58.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:58.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:15:59.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:15:59.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:00.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:00.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:01.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:01.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:02.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:02.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:03.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:03.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:04.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:04.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:05.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:05.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:06.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:06.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:07.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:07.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:08.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:08.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:09.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:09.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:10.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:10.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:11.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:11.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:12.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:12.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:13.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:13.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:14.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:14.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:15.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:15.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:16.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:16.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:17.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:17.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:18.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:18.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:19.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:19.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:20.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:20.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:21.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:21.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:22.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:22.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:23.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:23.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:24.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:24.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:25.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:25.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:26.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:26.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:27.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:27.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:28.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:28.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:29.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:29.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:30.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:30.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:31.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:31.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:32.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:32.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:33.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:33.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:34.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:34.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:35.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:35.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:36.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:36.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:36.800 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:16:36.800 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:16:36.801 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:16:36.801 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:36.801 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:16:36.807 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:16:36.807 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T06:16:36.808 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T06:16:36.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:16:37.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:37.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:38.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:38.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:39.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:39.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:40.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:40.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:41.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:41.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:42.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:42.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:43.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:43.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:44.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:44.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:45.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:45.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:46.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:46.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:47.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:47.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:48.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:48.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:49.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:49.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:50.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:50.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:51.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:51.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:51.812 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T06:16:51.813 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T06:16:51.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:16:51.813 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:51.813 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:16:52.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:52.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:52.829 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:16:52.851 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:16:52.852 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:16:52.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:52.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:16:53.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:53.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:53.866 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":990,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:16:53.898 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:53.898 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:16:54.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:54.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:54.888 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":990,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:16:54.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:54.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:16:55.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:55.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:55.912 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":990,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:16:55.935 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:16:55.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:55.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:16:56.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:56.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:57.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:16:57.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:57.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:58.031 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:16:58.031 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:58.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:16:58.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:58.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:16:59.016 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:16:59.037 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:16:59.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:16:59.038 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:16:59.038 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:16:59.038 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:16:59.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:16:59.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:00.025 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":990,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:17:00.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:00.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:17:00.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:00.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:01.033 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":990,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:17:01.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:01.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:17:01.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:01.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:02.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":990,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T06:17:02.096 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:17:02.096 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:17:02.096 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:17:02.097 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:17:02.097 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:17:02.097 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:17:02.139 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:17:02.139 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:17:02.139 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:02.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:17:02.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:02.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:03.127 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:03.149 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:03.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:03.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:04.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:04.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:05.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:05.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:06.151 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:06.151 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:17:06.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:06.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:07.149 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:07.182 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:07.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:07.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:08.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:08.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:09.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:09.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:10.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:10.183 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:17:10.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:10.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:11.200 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:11.221 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":990,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:11.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:11.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:12.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:12.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:13.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:13.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:14.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:14.223 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:17:14.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:14.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:15.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:15.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:16.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:16.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:17.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:17.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:17.755 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:17.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:17:17.778 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:17:17.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:17.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:17:18.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:18.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:18.672 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":992,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:17:18.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:17:18.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:17:19.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:19.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:19.667 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":992,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:17:19.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:17:20.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:20.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:21.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:21.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:22.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:22.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:23.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:23.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:24.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:24.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:25.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:25.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:26.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:26.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:27.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:27.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:28.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:28.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:29.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:29.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:30.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:30.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:31.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:31.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:32.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:32.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:33.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:33.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:34.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:34.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:35.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:35.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:36.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:36.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:37.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:37.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:38.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:38.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:39.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:39.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:40.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:40.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:41.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:41.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:42.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:42.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:43.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:43.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:44.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:44.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:45.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:45.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:46.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:46.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:47.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:47.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:48.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:48.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:49.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:49.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:50.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:50.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:51.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:51.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:52.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:52.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:53.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:53.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:54.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:54.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:55.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:55.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:56.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:56.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:57.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:57.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:58.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:58.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:17:59.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:17:59.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:00.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:00.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:01.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:01.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:02.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:02.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:03.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:03.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:04.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:04.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:05.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:05.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:06.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:06.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:07.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:07.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:08.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:08.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:09.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:09.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:10.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:10.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:11.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:11.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:12.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:12.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:13.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:13.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:14.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:14.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:15.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:15.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:16.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:16.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:17.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:17.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:18.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:18.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:19.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:19.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:20.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:20.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:21.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:21.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:22.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:22.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:23.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:23.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:24.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:24.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:25.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:25.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:26.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:26.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:27.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:27.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:28.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:28.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:29.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:29.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:30.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:30.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:31.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:31.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:32.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:32.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:33.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:33.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:34.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:34.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:35.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:35.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:36.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:36.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:37.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:37.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:38.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:38.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:39.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:39.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:40.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:40.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:41.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:41.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:42.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:42.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:43.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:43.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:44.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:44.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:45.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:45.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:46.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:46.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:47.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:47.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:48.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:48.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:49.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:49.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:49.690 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:18:49.690 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:18:49.690 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:18:49.690 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:18:49.690 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:18:49.696 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:18:49.696 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:18:49.696 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:18:49.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:18:50.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:50.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:50.708 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:50.730 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:51.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:51.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:52.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:52.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:53.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:53.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:53.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:18:53.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:18:54.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:54.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:54.729 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:54.760 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:55.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:55.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:56.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:56.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:57.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:57.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:57.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:18:57.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:18:58.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:58.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:18:58.783 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:58.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:18:59.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:18:59.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:00.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:00.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:01.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:01.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:01.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:01.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:02.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:02.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:02.816 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:02.844 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":992,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:03.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:03.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:04.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:04.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:05.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:05.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:05.845 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:05.845 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:06.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:06.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:06.842 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:06.872 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:06.873 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:19:06.873 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:06.873 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:19:07.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:07.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:07.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":996,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:07.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:07.857 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:19:08.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:08.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:08.854 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":996,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:08.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:08.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:19:09.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:09.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:09.889 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":996,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:09.911 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:19:09.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:09.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:19:10.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:10.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:10.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:19:11.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:11.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:11.987 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:19:11.987 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:11.987 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:12.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:12.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:12.974 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:12.995 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:12.995 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:19:12.996 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:19:12.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:12.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:19:13.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:13.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:13.947 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":996,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:13.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:13.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:19:14.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:14.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:14.953 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":996,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:14.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:14.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:19:15.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:15.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:15.948 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":996,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:15.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:19:15.979 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:19:15.979 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:19:15.979 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:19:15.979 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:19:15.979 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:19:15.998 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:19:15.998 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:19:15.999 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:15.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:16.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:16.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:16.991 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:17.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:17.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:17.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:18.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:18.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:19.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:19.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:20.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:20.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:20.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:20.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:21.000 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:21.025 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:21.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:21.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:22.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:22.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:23.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:23.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:24.025 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:24.026 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:24.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:24.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:25.005 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:25.026 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":996,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:25.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:25.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:26.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:26.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:27.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:27.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:28.027 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:28.027 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:19:28.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:28.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:29.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:29.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:30.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:30.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:30.918 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:30.950 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:19:30.950 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:19:30.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:30.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:19:31.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:31.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:31.948 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":998,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:31.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:19:31.979 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:19:32.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:32.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:33.234 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":998,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:19:33.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:19:33.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:33.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:34.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:34.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:35.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:35.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:36.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:36.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:37.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:37.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:38.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:38.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:39.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:39.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:40.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:40.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:41.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:41.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:42.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:42.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:43.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:43.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:44.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:44.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:45.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:45.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:46.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:46.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:47.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:47.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:48.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:48.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:49.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:49.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:51.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:51.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:51.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:51.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:53.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:53.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:54.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:54.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:55.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:55.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:56.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:56.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:57.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:57.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:58.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:58.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:19:59.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:19:59.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:00.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:00.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:01.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:01.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:02.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:02.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:03.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:03.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:04.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:04.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:05.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:05.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:06.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:06.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:07.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:07.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:07.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:07.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:08.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:08.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:09.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:09.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:10.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:10.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:11.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:11.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:12.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:12.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:13.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:13.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:14.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:14.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:15.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:15.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:16.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:16.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:17.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:17.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:18.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:18.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:19.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:19.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:20.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:20.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:21.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:21.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:22.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:22.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:23.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:23.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:25.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:25.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:26.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:26.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:27.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:27.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:28.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:28.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:29.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:29.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:29.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:29.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:30.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:30.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:31.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:31.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:32.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:32.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:34.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:34.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:35.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:35.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:36.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:36.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:37.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:37.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:38.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:38.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:39.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:39.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:40.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:40.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:41.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:41.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:42.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:42.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:43.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:43.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:44.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:44.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:44.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:44.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:46.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:46.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:47.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:47.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:48.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:48.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:49.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:49.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:50.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:50.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:51.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:51.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:52.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:52.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:52.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:52.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:53.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:53.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:55.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:54.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:55.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:55.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:56.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:56.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:57.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:57.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:20:58.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:20:58.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:00.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:00.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:01.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:00.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:01.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:01.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:02.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:02.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:03.266 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:21:03.267 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:21:03.267 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:21:03.267 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:21:03.267 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:21:03.275 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:21:03.275 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:21:03.275 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:03.275 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:03.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:03.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:04.289 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:04.312 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:04.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:04.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:05.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:05.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:06.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:06.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:07.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:07.312 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:07.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:07.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:08.326 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:08.349 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:08.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:08.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:09.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:09.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:10.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:10.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:11.350 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:11.350 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:11.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:11.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:12.331 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:12.360 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:12.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:12.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:13.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:13.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:14.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:14.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:15.361 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:15.361 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:15.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:15.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:16.377 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:16.401 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":998,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:16.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:16.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:18.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:17.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:18.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:18.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:19.402 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:19.402 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:19.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:19.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:20.388 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:20.410 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:20.410 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:21:20.411 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:20.411 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:21:20.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:20.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:21.419 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1002,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:21.451 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:21.451 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:21:21.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:21.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:22.450 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1002,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:22.475 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:22.475 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:21:22.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:22.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:23.445 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1002,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:23.475 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:21:23.475 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:23.475 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:21:23.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:23.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:24.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:21:24.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:24.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:25.570 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:21:25.570 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:25.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:25.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:25.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:26.586 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:26.618 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:26.618 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:21:26.618 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:21:26.619 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:26.619 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:21:26.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:26.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:27.615 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1002,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:27.638 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:27.638 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:21:27.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:27.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:28.665 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1002,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:28.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:28.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:21:28.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:28.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:29.639 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1002,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:29.661 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:21:29.661 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:21:29.662 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:21:29.662 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:21:29.662 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:21:29.662 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:21:29.683 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:21:29.684 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:21:29.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:29.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:29.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:29.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:30.676 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:30.713 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:30.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:30.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:31.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:31.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:32.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:32.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:33.714 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:33.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:33.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:33.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:34.740 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:34.762 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:35.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:35.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:36.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:36.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:37.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:37.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:37.763 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:37.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:38.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:38.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:38.760 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:38.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1002,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:39.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:39.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:40.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:40.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:41.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:41.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:41.783 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:41.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:21:42.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:42.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:43.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:43.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:43.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:43.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:44.889 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:44.912 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:21:44.912 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:21:44.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:44.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:21:45.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:45.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:45.888 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1006,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:45.919 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:21:45.919 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:21:46.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:46.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:46.915 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1006,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:21:46.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:21:46.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:46.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:47.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:47.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:48.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:48.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:49.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:49.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:50.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:50.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:51.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:51.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:52.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:52.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:53.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:53.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:54.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:54.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:55.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:55.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:56.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:56.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:57.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:57.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:58.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:58.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:21:59.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:21:59.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:00.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:00.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:01.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:01.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:02.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:02.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:03.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:03.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:04.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:04.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:05.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:05.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:06.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:06.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:07.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:07.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:08.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:08.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:09.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:09.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:10.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:10.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:11.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:11.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:12.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:12.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:13.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:13.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:14.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:14.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:15.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:15.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:16.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:16.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:17.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:17.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:18.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:18.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:19.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:19.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:20.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:20.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:21.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:21.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:22.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:22.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:23.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:23.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:25.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:24.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:25.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:25.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:27.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:27.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:28.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:28.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:29.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:29.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:29.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:29.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:31.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:31.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:32.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:31.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:33.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:33.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:34.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:34.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:35.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:35.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:36.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:36.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:37.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:37.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:37.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:37.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:38.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:38.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:39.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:39.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:40.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:40.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:41.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:41.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:42.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:42.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:43.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:43.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:44.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:44.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:45.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:45.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:46.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:46.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:47.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:47.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:48.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:48.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:49.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:49.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:50.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:50.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:51.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:51.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:52.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:52.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:53.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:53.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:54.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:54.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:55.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:55.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:57.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:57.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:58.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:58.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:22:59.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:22:59.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:00.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:00.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:01.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:01.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:02.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:02.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:03.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:03.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:04.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:04.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:05.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:05.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:06.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:06.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:07.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:07.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:08.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:08.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:08.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:08.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:09.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:09.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:10.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:10.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:11.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:11.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:12.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:12.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:13.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:13.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:14.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:14.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:15.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:15.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:16.938 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:23:16.938 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:23:16.939 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:23:16.939 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:16.939 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:23:16.945 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:23:16.945 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:23:16.945 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:16.945 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:16.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:16.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:17.924 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:17.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:17.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:17.951 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:18.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:18.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:19.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:19.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:20.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:20.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:20.952 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:20.952 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:21.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:21.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:21.978 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:21.999 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:22.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:22.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:23.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:23.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:24.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:24.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:25.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:25.001 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:25.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:25.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:26.015 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:26.041 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1006,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:26.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:26.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:27.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:27.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:28.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:28.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:29.041 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:29.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:29.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:29.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:30.030 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:30.054 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:30.054 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:23:30.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:30.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:23:30.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:30.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:31.053 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1008,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:31.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:31.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:23:31.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:31.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:32.082 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1008,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:32.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:32.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:23:32.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:32.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:33.094 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1008,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:33.116 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:23:33.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:33.117 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:23:33.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:33.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:34.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:23:34.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:34.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:35.222 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:23:35.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:35.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:35.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:35.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:36.241 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:36.263 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:36.263 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:23:36.264 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:23:36.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:36.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:23:36.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:36.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:37.247 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1008,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:37.271 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:37.271 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:23:37.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:37.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:38.282 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1008,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:38.303 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:38.303 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:23:38.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:38.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:39.292 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1008,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:39.314 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:23:39.315 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:23:39.315 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:23:39.315 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:23:39.315 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:23:39.315 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:23:39.336 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:23:39.336 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:23:39.336 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:39.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:39.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:39.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:40.371 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:40.400 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:40.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:40.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:41.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:41.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:42.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:42.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:43.401 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:43.402 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:43.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:43.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:44.408 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:44.431 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:44.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:44.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:45.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:45.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:46.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:46.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:47.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:47.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:47.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:47.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:48.402 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:48.425 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1008,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:48.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:48.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:49.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:49.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:50.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:50.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:51.426 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:51.426 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:23:51.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:51.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:52.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:52.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:53.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:53.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:54.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:54.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:55.448 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:55.479 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:23:55.479 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:23:55.479 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:55.479 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:23:55.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:55.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:56.473 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1012,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:56.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:23:56.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:23:56.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:56.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:57.458 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1012,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:23:57.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:23:57.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:57.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:58.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:58.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:23:59.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:23:59.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:00.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:00.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:01.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:01.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:02.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:02.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:03.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:03.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:04.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:04.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:05.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:05.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:06.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:06.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:07.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:07.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:08.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:08.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:09.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:09.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:10.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:10.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:11.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:11.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:12.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:12.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:13.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:13.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:14.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:14.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:15.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:15.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:16.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:16.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:17.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:17.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:18.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:18.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:19.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:19.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:20.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:20.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:21.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:21.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:22.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:22.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:23.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:23.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:24.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:24.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:25.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:25.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:26.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:26.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:27.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:27.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:28.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:28.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:29.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:29.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:30.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:30.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:31.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:31.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:32.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:32.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:33.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:33.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:34.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:34.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:35.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:35.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:36.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:36.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:37.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:37.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:38.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:38.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:39.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:39.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:40.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:40.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:41.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:41.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:42.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:42.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:43.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:43.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:44.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:44.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:45.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:45.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:46.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:46.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:47.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:47.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:48.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:48.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:49.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:49.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:50.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:50.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:51.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:51.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:52.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:52.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:53.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:53.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:54.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:54.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:55.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:55.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:56.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:56.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:57.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:57.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:58.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:58.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:24:59.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:24:59.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:00.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:00.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:01.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:01.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:02.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:02.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:03.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:03.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:04.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:04.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:05.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:05.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:06.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:06.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:07.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:07.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:08.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:08.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:09.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:09.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:10.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:10.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:11.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:11.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:12.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:12.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:14.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:13.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:15.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:15.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:16.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:16.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:16.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:16.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:17.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:17.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:18.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:18.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:19.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:19.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:20.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:20.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:21.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:21.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:22.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:22.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:24.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:23.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:24.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:24.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:25.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:25.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:26.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:26.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:27.481 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:25:27.481 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:25:27.482 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:25:27.482 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:27.482 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:25:27.488 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:25:27.488 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:25:27.488 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:27.489 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:27.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:27.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:28.445 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:28.467 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:28.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:28.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:29.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:29.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:30.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:30.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:31.468 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:31.468 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:31.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:31.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:32.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:32.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:33.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:33.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:34.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:34.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:35.725 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:35.747 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:35.747 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:25:35.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:35.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:25:35.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:35.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:36.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:36.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:36.763 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1018,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:36.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:36.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:25:37.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:37.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:38.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:38.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:39.488 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1018,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:39.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:39.510 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:25:39.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:39.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:40.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1018,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:40.557 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:25:40.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:40.557 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:25:40.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:40.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:41.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:25:41.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:41.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:42.656 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:25:42.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:42.657 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:42.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:42.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:43.639 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:43.661 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:43.662 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:25:43.662 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:25:43.662 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:43.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:25:43.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:43.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:44.691 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1018,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:44.714 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:44.714 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:25:44.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:44.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:45.698 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1018,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:45.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:45.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:25:45.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:45.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:46.708 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1018,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:25:46.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:25:46.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:25:46.733 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:25:46.733 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:25:46.733 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:25:46.733 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:25:46.752 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:25:46.752 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:25:46.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:46.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:46.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:46.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:47.821 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:47.852 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:47.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:47.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:48.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:48.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:49.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:49.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:50.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:50.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:50.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:50.853 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:51.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:51.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:52.085 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:52.108 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:52.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:52.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:53.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:53.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:54.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:54.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:55.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:55.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:55.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:55.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:56.092 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:56.123 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1018,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:25:56.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:56.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:57.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:57.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:58.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:58.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:25:59.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:25:59.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:25:59.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:25:59.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:00.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:00.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:01.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:01.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:01.754 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:26:01.776 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:26:01.776 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:26:01.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:26:01.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:26:02.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:02.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:02.790 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1022,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:26:02.822 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:26:02.822 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:26:03.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:03.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:03.836 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1022,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:26:03.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:26:04.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:04.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:05.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:05.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:06.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:06.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:07.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:07.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:08.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:08.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:09.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:09.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:10.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:10.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:11.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:11.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:12.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:12.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:13.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:13.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:14.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:14.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:15.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:15.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:16.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:16.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:17.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:17.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:18.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:18.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:19.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:19.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:20.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:20.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:21.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:21.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:22.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:22.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:23.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:23.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:24.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:24.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:25.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:25.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:26.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:26.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:27.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:27.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:28.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:28.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:29.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:29.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:30.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:30.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:31.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:31.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:32.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:32.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:33.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:33.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:34.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:34.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:35.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:35.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:36.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:36.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:37.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:37.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:38.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:38.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:39.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:39.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:40.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:40.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:41.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:41.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:42.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:42.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:43.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:43.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:44.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:44.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:45.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:45.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:46.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:46.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:47.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:47.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:48.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:48.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:49.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:49.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:50.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:50.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:51.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:51.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:52.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:52.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:53.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:53.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:54.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:54.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:55.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:55.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:56.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:56.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:57.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:57.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:58.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:58.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:26:59.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:26:59.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:00.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:00.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:01.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:01.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:02.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:02.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:03.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:03.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:04.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:04.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:05.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:05.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:06.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:06.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:07.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:07.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:08.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:08.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:09.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:09.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:10.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:10.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:11.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:11.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:12.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:12.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:13.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:13.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:14.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:14.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:15.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:15.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:16.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:16.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:17.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:17.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:18.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:18.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:19.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:19.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:20.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:20.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:21.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:21.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:22.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:22.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:23.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:23.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:24.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:24.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:25.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:25.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:26.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:26.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:27.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:27.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:28.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:28.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:29.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:29.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:30.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:30.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:31.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:31.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:32.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:32.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:33.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:33.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:33.865 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:27:33.865 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:27:33.866 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:27:33.866 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:33.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:27:33.874 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:27:33.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:27:33.874 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:33.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:27:34.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:34.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:34.836 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:34.859 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:35.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:35.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:36.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:36.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:37.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:37.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:37.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:37.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:27:38.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:38.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:38.869 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:38.890 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:39.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:39.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:40.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:40.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:41.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:41.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:41.891 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:41.891 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:27:42.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:42.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:42.902 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:42.928 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1022,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:43.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:43.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:44.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:44.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:45.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:45.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:45.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:45.929 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:27:46.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:46.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:46.946 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1024,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:46.978 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1024,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:46.978 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:27:46.978 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:46.978 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:27:47.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:47.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:47.986 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1024,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:48.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:48.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:27:48.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:48.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:49.041 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1024,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:49.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:49.072 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:27:49.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:49.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:50.064 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1024,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:50.086 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:27:50.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:50.086 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:27:50.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:50.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:51.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:27:51.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:51.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:52.163 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:27:52.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:52.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:27:52.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:52.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:53.170 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1024,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:53.192 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1024,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:27:53.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:27:53.192 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:27:53.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:53.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:27:53.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:53.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:54.152 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1024,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:54.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:54.175 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:27:54.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:54.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:55.185 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1024,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:55.206 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:27:55.206 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:27:55.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:55.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:56.208 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1024,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:27:56.231 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:27:56.231 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T06:27:56.231 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:27:56.231 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:27:56.231 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:27:56.232 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:27:56.249 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:27:56.249 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:27:56.250 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:27:56.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:27:56.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:56.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:57.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:57.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:58.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:58.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:27:59.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:27:59.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:00.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:00.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:01.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:01.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:02.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:02.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:03.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:03.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:04.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:04.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:05.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:05.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:06.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:06.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:07.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:07.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:08.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:08.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:09.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:09.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:10.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:10.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:11.251 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:28:11.252 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:28:11.252 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:28:11.252 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:28:11.252 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:28:11.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:11.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:12.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:12.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:13.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:13.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:14.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:14.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:15.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:15.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:16.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:16.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:16.909 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:28:16.939 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:28:16.940 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:28:16.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:28:16.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:28:17.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:17.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:17.948 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1026,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:28:17.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:28:17.969 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:28:18.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:18.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:18.969 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1026,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:28:18.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:28:19.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:19.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:20.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:20.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:21.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:21.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:22.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:22.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:23.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:23.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:24.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:24.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:25.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:25.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:26.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:26.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:27.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:27.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:28.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:28.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:29.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:29.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:30.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:30.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:31.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:31.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:32.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:32.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:33.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:33.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:34.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:34.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:35.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:35.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:36.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:36.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:37.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:37.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:38.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:38.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:39.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:39.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:40.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:40.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:41.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:41.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:42.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:42.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:43.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:43.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:44.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:44.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:45.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:45.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:46.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:46.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:47.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:47.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:48.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:48.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:49.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:49.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:50.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:50.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:51.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:51.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:52.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:52.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:53.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:53.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:54.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:54.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:55.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:55.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:56.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:56.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:57.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:57.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:58.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:58.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:28:59.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:28:59.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:00.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:00.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:01.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:01.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:02.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:02.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:03.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:03.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:04.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:04.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:05.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:05.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:06.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:06.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:07.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:07.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:08.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:08.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:09.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:09.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:10.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:10.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:11.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:11.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:12.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:12.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:13.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:13.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:14.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:14.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:15.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:15.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:16.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:16.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:17.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:17.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:18.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:18.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:19.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:19.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:20.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:20.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:21.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:21.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:22.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:22.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:23.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:23.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:24.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:24.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:25.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:25.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:26.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:26.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:27.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:27.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:28.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:28.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:29.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:29.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:30.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:30.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:31.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:31.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:32.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:32.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:33.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:33.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:34.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:34.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:35.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:35.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:36.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:36.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:37.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:37.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:38.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:38.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:39.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:39.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:40.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:40.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:41.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:41.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:42.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:42.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:43.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:43.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:44.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:44.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:45.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:45.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:46.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:46.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:47.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:47.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:48.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:48.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:48.992 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:29:48.993 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:29:48.993 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:29:48.993 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:29:48.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:29:48.999 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:29:48.999 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:29:48.999 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:29:48.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:29:49.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:49.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:50.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:50.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:51.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:51.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:52.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:52.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:53.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:53.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:54.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:54.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:55.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:55.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:56.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:56.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:57.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:57.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:58.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:58.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:29:59.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:29:59.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:00.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:00.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:01.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:01.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:02.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:02.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:03.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:03.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:04.000 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:30:04.001 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:30:04.001 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:30:04.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:04.001 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:30:04.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:04.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:05.023 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:05.046 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:05.046 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:30:05.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:05.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:30:05.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:05.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:06.052 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1032,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:06.085 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:06.085 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:30:06.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:06.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:07.082 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1032,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:07.112 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:07.112 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:30:07.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:07.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:08.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:08.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:08.148 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1032,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:08.171 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:30:08.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:08.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:30:09.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:09.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:09.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:30:10.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:10.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:10.301 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:30:10.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:10.302 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:30:11.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:11.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:11.288 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:11.320 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:11.320 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:30:11.320 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:30:11.321 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:11.321 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:30:12.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:12.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:12.352 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1032,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:12.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:12.375 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:30:13.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:13.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:13.356 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1032,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:13.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:13.379 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:30:14.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:14.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:14.382 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1032,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:14.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:30:14.404 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T06:30:14.404 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:30:14.404 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:30:14.405 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:30:14.405 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:30:14.423 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:30:14.423 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:30:14.424 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:30:14.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:30:15.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:15.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:16.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:16.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:17.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:17.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:18.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:18.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:19.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:19.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:20.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:20.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:21.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:21.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:22.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:22.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:23.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:23.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:24.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:24.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:25.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:25.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:26.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:26.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:27.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:27.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:28.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:28.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:29.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:29.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:29.424 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:30:29.425 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:30:29.425 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:30:29.425 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:29.425 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:30:30.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:30.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:31.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:31.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:32.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:32.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:33.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:33.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:34.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:34.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:35.083 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1036,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:35.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1036,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:30:35.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:30:35.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:35.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:30:35.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:35.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:36.123 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1036,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:36.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:30:36.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:30:36.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:36.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:37.139 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1036,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:30:37.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:30:37.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:37.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:38.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:38.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:39.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:39.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:40.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:40.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:41.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:41.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:42.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:42.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:43.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:43.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:44.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:44.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:45.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:45.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:46.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:46.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:47.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:47.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:48.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:48.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:49.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:49.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:50.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:50.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:51.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:51.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:52.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:52.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:53.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:53.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:54.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:54.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:55.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:55.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:56.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:56.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:57.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:57.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:58.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:58.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:30:59.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:30:59.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:00.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:00.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:01.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:01.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:01.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:01.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:03.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:03.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:04.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:04.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:05.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:05.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:06.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:06.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:07.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:07.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:08.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:08.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:09.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:09.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:10.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:10.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:10.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:10.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:12.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:12.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:13.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:13.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:14.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:14.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:14.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:14.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:15.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:15.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:16.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:16.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:17.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:17.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:18.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:18.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:19.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:19.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:20.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:20.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:21.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:21.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:22.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:22.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:23.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:23.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:24.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:24.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:25.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:25.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:26.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:26.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:27.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:27.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:28.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:28.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:29.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:29.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:30.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:30.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:31.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:31.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:32.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:32.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:33.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:33.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:35.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:35.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:36.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:36.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:36.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:36.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:38.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:38.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:39.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:39.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:40.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:40.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:41.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:41.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:42.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:42.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:43.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:43.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:44.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:44.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:45.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:45.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:46.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:46.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:47.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:47.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:48.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:48.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:49.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:49.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:50.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:50.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:51.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:51.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:52.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:52.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:53.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:53.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:54.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:54.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:55.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:55.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:56.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:56.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:57.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:57.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:58.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:58.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:31:59.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:31:59.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:00.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:00.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:01.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:01.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:02.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:02.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:03.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:03.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:04.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:04.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:05.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:05.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:06.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:06.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:07.163 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:32:07.163 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:32:07.163 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:32:07.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:07.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:32:07.170 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:32:07.171 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:32:07.171 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:32:07.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:32:07.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:07.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:08.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:08.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:09.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:09.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:10.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:10.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:11.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:11.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:12.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:12.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:13.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:13.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:14.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:14.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:15.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:15.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:16.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:16.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:17.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:17.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:18.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:18.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:19.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:19.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:20.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:20.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:21.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:21.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:22.172 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:32:22.172 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:32:22.173 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:32:22.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:22.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:22.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:22.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:23.176 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:23.198 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:23.198 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:32:23.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:23.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:32:23.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:23.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:24.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1040,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:24.193 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:24.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:32:24.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:24.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:25.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:25.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:25.401 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1040,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:25.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:25.430 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:32:26.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:26.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:26.399 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1040,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:26.421 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:32:26.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:26.422 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:32:27.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:27.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:27.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:32:28.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:28.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:28.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:32:28.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:28.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:29.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:29.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:29.538 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:29.570 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:29.570 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:32:29.570 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:32:29.571 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:29.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:32:30.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:30.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:30.605 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1040,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:30.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:30.627 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:32:31.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:31.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:31.614 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1040,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:31.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:31.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:32:32.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:32.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:32.600 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1040,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:32.621 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:32:32.621 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:32:32.621 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:32:32.622 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:32:32.622 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:32:32.622 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:32:32.636 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:32:32.636 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:32:32.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:32.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:33.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:33.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:33.705 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:33.722 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:34.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:34.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:35.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:35.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:36.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:36.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:36.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:36.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:37.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:37.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:37.727 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:37.759 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:38.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:38.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:39.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:39.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:40.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:40.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:40.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:40.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:41.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:41.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:41.744 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:41.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1040,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:42.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:42.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:43.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:43.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:44.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:44.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:44.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:44.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:32:45.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:45.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:46.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:46.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:47.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:47.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:48.166 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1044,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:48.188 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1044,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:32:48.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:32:48.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:48.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:32:48.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:48.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:49.169 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1044,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:49.193 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:32:49.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:32:49.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:49.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:50.174 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1044,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:32:50.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:32:50.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:50.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:51.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:51.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:52.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:52.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:53.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:53.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:54.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:54.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:55.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:55.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:56.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:56.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:57.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:57.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:58.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:58.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:32:59.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:32:59.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:00.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:00.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:01.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:01.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:02.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:02.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:03.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:03.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:04.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:04.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:05.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:05.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:06.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:06.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:07.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:07.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:08.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:08.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:09.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:09.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:10.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:10.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:11.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:11.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:12.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:12.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:13.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:13.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:14.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:14.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:15.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:15.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:16.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:16.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:17.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:17.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:18.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:18.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:19.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:19.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:20.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:20.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:21.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:21.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:22.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:22.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:23.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:23.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:24.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:24.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:25.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:25.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:26.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:26.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:27.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:27.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:28.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:28.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:29.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:29.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:30.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:30.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:31.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:31.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:32.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:32.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:33.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:33.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:34.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:34.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:35.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:35.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:36.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:36.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:37.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:37.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:38.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:38.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:39.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:39.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:40.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:40.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:41.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:41.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:42.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:42.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:43.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:43.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:44.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:44.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:45.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:45.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:46.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:46.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:47.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:47.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:48.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:48.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:49.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:49.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:50.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:50.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:51.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:51.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:52.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:52.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:53.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:53.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:54.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:54.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:55.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:55.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:56.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:56.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:57.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:57.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:58.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:58.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:33:59.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:33:59.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:00.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:00.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:01.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:01.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:02.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:02.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:03.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:03.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:04.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:04.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:05.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:05.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:06.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:06.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:07.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:07.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:08.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:08.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:09.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:09.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:10.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:10.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:11.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:11.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:12.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:12.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:13.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:13.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:14.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:14.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:15.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:15.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:16.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:16.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:17.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:17.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:18.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:18.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:19.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:19.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:20.197 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:34:20.198 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:34:20.198 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:34:20.198 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:20.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:34:20.205 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:34:20.206 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:34:20.206 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:20.206 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:20.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:20.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:21.161 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1044,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:21.184 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1044,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:21.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:21.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:22.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:22.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:23.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:23.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:24.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:24.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:24.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:24.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:25.188 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:25.217 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:25.218 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:34:25.218 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:25.218 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:34:25.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:25.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:26.218 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1046,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:26.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:26.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:34:26.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:26.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:27.258 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1046,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:27.289 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:27.289 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:34:27.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:27.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:28.304 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1046,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:28.334 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:34:28.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:28.335 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:34:28.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:28.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:29.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:29.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:29.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:34:30.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:30.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:30.427 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:34:30.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:30.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:31.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:31.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:31.429 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:31.459 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:31.459 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:34:31.459 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:34:31.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:31.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:34:32.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:32.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:32.477 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1046,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:32.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:32.500 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:34:33.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:33.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:33.498 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1046,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:33.519 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:33.519 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:34:34.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:34.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:34.452 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1046,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:34.473 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:34:34.474 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:34:34.474 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:34:34.474 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:34:34.474 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:34:34.474 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:34:34.500 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:34:34.500 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:34:34.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:34.500 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:35.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:35.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:35.533 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:35.556 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:36.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:36.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:37.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:37.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:38.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:38.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:38.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:38.557 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:39.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:39.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:39.562 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:39.580 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:40.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:40.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:41.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:41.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:42.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:42.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:42.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:42.581 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:43.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:43.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:43.568 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:43.598 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1046,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:44.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:44.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:45.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:45.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:46.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:46.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:46.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:46.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:34:47.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:47.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:48.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:48.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:49.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:49.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:49.959 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:49.980 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:34:49.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:34:49.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:49.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:34:50.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:50.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:50.990 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1050,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:51.012 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:34:51.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:34:51.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:51.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:51.990 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1050,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:34:52.013 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:34:52.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:52.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:53.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:53.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:54.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:54.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:55.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:55.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:56.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:56.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:57.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:57.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:58.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:58.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:34:59.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:34:59.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:00.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:00.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:01.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:01.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:02.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:02.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:03.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:03.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:04.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:04.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:05.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:05.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:06.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:06.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:07.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:07.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:08.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:08.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:09.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:09.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:10.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:10.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:11.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:11.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:12.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:12.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:13.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:13.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:14.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:14.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:15.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:15.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:16.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:16.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:17.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:17.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:18.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:18.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:19.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:19.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:20.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:20.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:21.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:21.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:22.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:22.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:23.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:23.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:24.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:24.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:25.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:25.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:26.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:26.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:27.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:27.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:28.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:28.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:29.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:29.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:30.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:30.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:31.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:31.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:32.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:32.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:33.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:33.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:34.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:34.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:35.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:35.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:36.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:36.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:37.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:37.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:38.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:38.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:39.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:39.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:40.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:40.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:41.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:41.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:42.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:42.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:43.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:43.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:44.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:44.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:45.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:45.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:46.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:46.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:47.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:47.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:48.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:48.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:49.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:49.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:50.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:50.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:51.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:51.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:52.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:52.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:53.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:53.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:54.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:54.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:55.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:55.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:56.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:56.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:57.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:57.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:58.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:58.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:35:59.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:35:59.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:00.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:00.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:01.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:01.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:02.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:02.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:03.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:03.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:04.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:04.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:05.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:05.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:06.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:06.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:07.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:07.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:08.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:08.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:09.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:09.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:10.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:10.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:11.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:11.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:12.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:12.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:13.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:13.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:14.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:14.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:15.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:15.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:16.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:16.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:17.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:17.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:18.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:18.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:19.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:19.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:20.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:20.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:21.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:21.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:22.014 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:36:22.014 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:36:22.015 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:36:22.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:22.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:36:22.021 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:36:22.021 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:36:22.021 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:22.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:22.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:22.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:23.007 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:23.029 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:23.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:23.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:24.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:24.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:25.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:25.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:26.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:26.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:26.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:26.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:27.041 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:27.066 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:27.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:27.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:28.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:28.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:29.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:29.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:30.067 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:30.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:30.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:30.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:31.110 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:31.134 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1050,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:31.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:31.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:32.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:32.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:33.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:33.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:34.135 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:34.135 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:34.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:34.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:35.122 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:35.144 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:35.145 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:36:35.145 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:35.145 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:36:35.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:35.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:36.144 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1052,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:36.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:36.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:36:36.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:36.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:37.164 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1052,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:37.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:37.197 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:36:37.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:37.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:38.184 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1052,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:38.214 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:36:38.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:38.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:36:38.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:38.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:39.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:36:39.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:39.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:40.298 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:36:40.299 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:40.299 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:40.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:40.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:41.308 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:41.338 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:41.338 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:36:41.338 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:36:41.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:41.339 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:36:41.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:41.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:42.309 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1052,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:42.341 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:42.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:36:42.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:42.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:43.301 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1052,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:43.324 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:43.324 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:36:43.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:43.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:44.304 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1052,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:36:44.328 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:36:44.328 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:36:44.328 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:36:44.329 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:36:44.329 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:36:44.329 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:36:44.345 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:36:44.345 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:36:44.345 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:44.345 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:44.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:44.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:45.384 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:45.414 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:45.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:45.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:46.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:46.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:47.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:47.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:48.415 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:48.416 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:48.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:48.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:49.438 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:49.461 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:49.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:49.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:50.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:50.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:51.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:51.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:52.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:52.463 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:52.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:52.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:53.468 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:53.498 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1052,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:53.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:53.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:54.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:54.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:55.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:55.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:56.499 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:56.499 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:36:56.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:56.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:57.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:57.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:58.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:58.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:59.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:36:59.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:36:59.932 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1056,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:59.962 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1056,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:36:59.962 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:36:59.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:36:59.963 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:37:00.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:00.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:01.002 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1056,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:37:01.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:37:01.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:37:01.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:01.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:01.988 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1056,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:37:02.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:37:02.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:02.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:03.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:03.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:04.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:04.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:05.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:05.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:06.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:06.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:07.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:07.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:08.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:08.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:09.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:09.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:10.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:10.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:11.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:11.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:12.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:12.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:13.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:13.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:14.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:14.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:15.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:15.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:16.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:16.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:17.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:17.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:18.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:18.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:19.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:19.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:20.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:20.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:21.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:21.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:22.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:22.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:23.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:23.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:24.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:24.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:25.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:25.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:26.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:26.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:27.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:27.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:28.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:28.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:29.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:29.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:30.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:30.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:31.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:31.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:32.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:32.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:33.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:33.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:34.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:34.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:35.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:35.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:36.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:36.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:37.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:37.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:38.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:38.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:39.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:39.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:40.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:40.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:41.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:41.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:42.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:42.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:43.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:43.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:44.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:44.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:45.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:45.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:46.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:46.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:47.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:47.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:48.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:48.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:49.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:49.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:50.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:50.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:51.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:51.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:52.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:52.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:53.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:53.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:54.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:54.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:55.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:55.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:56.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:56.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:57.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:57.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:58.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:58.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:37:59.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:37:59.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:00.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:00.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:01.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:01.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:02.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:02.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:03.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:03.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:04.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:04.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:05.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:05.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:06.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:06.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:07.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:07.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:08.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:08.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:09.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:09.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:10.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:10.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:11.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:11.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:12.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:12.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:13.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:13.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:14.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:14.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:15.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:15.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:16.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:16.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:17.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:17.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:18.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:18.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:19.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:19.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:20.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:20.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:21.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:21.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:22.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:22.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:23.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:23.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:24.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:24.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:25.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:25.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:26.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:26.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:27.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:27.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:28.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:28.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:29.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:29.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:30.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:30.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:31.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:31.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:32.019 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:38:32.019 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:38:32.019 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:38:32.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:32.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:38:32.028 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:38:32.028 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:38:32.028 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:32.028 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:38:32.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:32.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:33.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:33.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:34.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:34.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:35.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:35.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:36.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:36.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:37.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:37.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:37.802 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1060,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:37.834 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1060,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:38.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:38.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:39.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:39.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:40.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:40.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:40.835 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:40.835 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:38:41.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:41.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:42.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:42.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:43.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:43.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:43.423 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:43.453 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:43.453 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:38:43.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:43.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:38:44.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:44.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:44.467 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1068,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:44.497 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:44.497 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:38:45.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:45.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:45.489 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1068,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:45.512 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:45.512 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:38:46.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:46.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:46.529 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1068,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:46.561 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:38:46.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:46.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:38:47.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:47.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:47.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:38:48.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:48.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:48.709 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:38:48.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:48.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:38:49.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:49.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:49.695 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:49.717 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:49.717 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:38:49.718 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:38:49.718 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:49.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:38:50.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:50.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:50.674 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1068,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:50.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:50.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:38:51.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:51.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:51.691 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1068,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:51.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:51.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:38:52.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:52.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:52.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1068,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:38:52.749 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:38:52.749 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:38:52.749 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:38:52.750 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:38:52.750 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:38:52.750 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:38:52.788 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:38:52.789 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:38:52.789 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:52.789 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:38:53.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:53.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:53.786 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:53.816 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:54.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:54.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:55.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:55.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:56.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:56.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:56.817 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:38:56.817 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:38:57.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:57.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:57.804 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:57.826 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:38:58.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:58.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:38:59.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:38:59.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:00.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:00.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:00.826 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:39:00.826 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:39:01.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:01.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:01.848 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:39:01.876 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1068,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:39:02.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:02.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:03.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:03.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:04.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:04.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:04.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:39:04.877 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:39:05.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:05.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:06.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:06.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:07.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:07.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:08.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:08.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:09.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:09.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:10.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:10.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:11.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:11.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:12.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:12.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:13.017 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:39:13.047 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:39:13.047 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:39:13.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:39:13.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:39:13.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:13.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:14.037 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1072,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:39:14.067 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:39:14.067 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:39:14.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:14.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:15.037 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1072,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:39:15.068 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:39:15.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:15.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:16.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:16.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:17.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:17.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:18.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:18.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:19.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:19.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:20.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:20.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:21.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:21.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:22.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:22.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:23.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:23.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:24.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:24.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:25.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:25.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:26.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:26.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:27.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:27.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:28.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:28.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:29.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:29.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:30.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:30.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:31.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:31.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:32.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:32.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:33.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:33.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:34.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:34.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:35.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:35.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:36.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:36.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:37.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:37.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:38.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:38.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:39.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:39.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:40.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:40.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:41.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:41.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:42.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:42.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:43.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:43.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:44.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:44.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:45.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:45.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:46.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:46.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:47.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:47.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:48.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:48.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:49.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:49.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:50.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:50.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:51.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:51.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:52.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:52.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:53.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:53.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:54.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:54.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:55.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:55.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:56.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:56.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:57.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:57.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:58.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:58.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:39:59.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:39:59.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:00.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:00.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:01.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:01.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:02.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:02.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:03.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:03.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:04.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:04.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:05.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:05.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:06.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:06.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:07.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:07.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:08.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:08.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:09.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:09.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:10.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:10.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:11.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:11.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:12.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:12.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:13.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:13.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:14.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:14.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:15.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:15.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:16.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:16.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:17.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:17.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:18.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:18.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:19.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:19.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:20.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:20.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:21.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:21.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:22.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:22.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:23.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:23.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:24.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:24.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:25.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:25.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:26.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:26.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:27.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:27.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:28.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:28.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:29.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:29.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:30.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:30.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 70 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:31.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:31.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:32.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:32.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:33.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:33.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:34.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:34.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:35.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:35.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:36.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:36.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:37.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:37.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:38.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:38.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:39.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:39.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:40.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:40.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:41.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:41.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:42.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:42.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:43.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:43.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:44.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:44.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:45.069 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:40:45.069 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:40:45.069 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:40:45.069 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:40:45.070 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:40:45.096 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:40:45.097 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:40:45.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:45.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:40:45.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:45.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:46.117 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:46.138 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:46.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:46.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:47.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:47.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:48.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:48.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:49.139 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:49.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:40:49.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:49.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:50.146 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1076,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:50.177 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1076,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:50.177 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:40:50.178 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:50.178 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:40:50.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:50.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:51.162 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1076,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:51.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:51.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:40:51.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:51.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:52.189 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1076,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:52.211 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:52.211 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:40:52.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:52.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:53.173 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1076,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:53.195 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:40:53.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:53.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:40:53.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:53.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:54.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:40:54.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:54.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:55.314 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:40:55.314 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:55.314 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:40:55.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:55.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:56.337 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1076,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:56.370 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1076,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:40:56.370 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:40:56.370 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:40:56.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:56.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:40:56.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:56.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:57.385 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1076,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:57.416 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:57.416 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:40:57.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:57.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:58.427 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1076,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:58.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:40:58.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:40:58.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:58.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:40:59.421 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1076,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:40:59.446 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:40:59.446 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T06:40:59.446 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:40:59.446 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:40:59.447 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:40:59.447 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:40:59.467 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:40:59.467 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:40:59.467 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:40:59.467 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:40:59.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:40:59.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:00.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:00.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:01.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:01.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:02.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:02.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:03.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:03.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:04.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:04.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:05.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:05.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:06.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:06.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:07.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:07.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:08.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:08.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:09.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:09.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:10.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:10.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:11.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:11.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:12.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:12.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:13.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:13.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:14.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:14.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:14.468 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:41:14.468 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:41:14.469 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:41:14.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:41:14.469 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:41:15.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:15.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:16.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:16.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:17.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:17.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:18.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:18.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:19.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:19.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:20.111 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1078,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:41:20.140 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1078,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:41:20.140 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:41:20.140 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:41:20.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:41:20.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:20.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:21.128 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1078,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:41:21.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:41:21.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:41:21.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:21.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:22.160 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1078,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:41:22.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:41:22.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:22.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:23.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:23.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:24.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:24.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:25.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:25.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:26.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:26.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:27.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:27.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:28.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:28.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:29.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:29.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:30.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:30.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:31.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:31.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:32.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:32.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:33.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:33.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:34.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:34.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:35.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:35.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:36.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:36.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:37.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:37.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:38.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:38.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:39.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:39.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:40.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:40.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:41.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:41.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:42.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:42.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:43.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:43.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:44.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:44.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:45.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:45.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:46.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:46.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:47.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:47.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:48.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:48.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:49.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:49.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:50.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:50.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:51.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:51.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:52.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:52.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:53.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:53.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:54.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:54.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:55.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:55.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:56.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:56.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:57.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:57.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:58.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:58.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:41:59.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:41:59.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:00.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:00.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:01.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:01.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:02.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:02.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:03.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:03.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:04.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:04.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:05.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:05.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:06.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:06.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:07.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:07.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:08.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:08.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:09.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:09.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:10.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:10.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:11.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:11.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:12.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:12.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:13.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:13.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:14.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:14.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:15.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:15.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:16.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:16.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:17.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:17.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:18.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:18.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:19.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:19.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:20.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:20.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:21.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:21.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:22.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:22.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:23.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:23.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:24.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:24.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:25.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:25.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:26.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:26.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:27.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:27.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:28.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:28.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:29.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:29.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:30.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:30.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:31.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:31.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:32.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:32.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:33.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:33.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:34.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:34.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:35.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:35.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:36.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:36.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:37.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:37.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:38.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:38.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:39.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:39.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:40.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:40.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:41.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:41.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:42.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:42.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:43.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:43.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:44.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:44.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:45.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:45.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:46.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:46.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:47.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:47.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:48.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:48.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:49.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:49.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:50.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:50.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:51.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:51.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:52.191 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:42:52.191 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:42:52.191 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:42:52.192 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:42:52.192 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:42:52.198 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:42:52.198 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:42:52.198 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:42:52.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:42:52.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:52.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:53.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:53.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:54.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:54.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:55.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:55.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:56.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:56.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:57.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:57.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:58.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:58.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:42:59.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:42:59.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:00.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:00.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:01.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:01.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:02.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:02.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:03.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:03.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:04.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:04.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:05.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:05.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:06.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:06.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:07.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:07.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:07.199 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:43:07.200 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:43:07.200 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:43:07.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:07.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:43:08.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:08.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:08.413 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:08.435 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:08.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:43:08.435 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:08.435 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:43:09.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:09.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:09.407 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1086,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:09.434 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:09.434 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:43:10.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:10.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:10.440 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1086,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:10.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:10.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:43:11.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:11.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:11.458 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1086,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:11.478 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:43:11.478 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:11.479 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:43:12.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:12.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:12.634 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:43:13.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:13.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:13.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:43:13.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:13.635 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:43:14.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:14.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:14.606 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:14.627 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:14.628 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:43:14.628 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:43:14.628 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:14.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:43:15.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:15.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:15.644 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1086,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:15.671 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:15.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:43:16.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:16.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:16.693 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1086,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:16.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:16.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:43:16.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:16.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:17.737 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1086,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:17.757 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:43:17.759 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T06:43:17.759 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:43:17.760 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:43:17.760 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:43:17.760 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:43:17.780 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:43:17.780 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:43:17.780 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:43:17.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:43:17.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:17.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:18.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:18.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:19.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:19.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:20.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:20.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:22.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:22.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:23.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:23.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:24.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:24.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:25.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:25.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:26.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:26.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:27.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:27.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:28.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:28.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:29.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:29.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:30.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:30.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:31.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:31.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:32.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:32.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:32.781 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:43:32.782 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:43:32.782 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:43:32.782 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:32.782 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:43:33.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:33.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:34.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:34.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:35.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:35.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:36.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:36.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:37.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:37.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:38.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:38.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:38.428 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1088,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:38.452 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1088,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:43:38.452 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:43:38.452 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:38.452 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:43:39.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:39.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:39.442 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1088,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:39.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:43:39.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:43:40.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:40.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:40.484 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1088,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:43:40.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:43:41.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:41.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:42.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:42.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:43.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:43.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:44.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:44.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:45.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:45.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:46.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:46.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:47.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:47.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:48.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:48.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:49.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:49.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:50.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:50.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:51.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:51.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:52.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:52.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:53.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:53.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:54.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:54.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:55.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:55.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:56.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:56.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:57.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:57.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:58.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:58.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:43:59.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:43:59.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:00.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:00.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:01.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:01.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:02.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:02.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:03.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:03.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:04.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:04.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:05.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:05.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:06.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:06.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:07.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:07.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:08.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:08.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:09.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:09.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:10.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:10.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:11.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:11.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:12.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:12.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:13.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:13.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:14.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:14.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:15.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:15.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:16.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:16.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:17.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:17.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:18.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:18.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:19.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:19.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:20.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:20.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:21.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:21.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:22.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:22.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:23.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:23.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:23.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:23.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:24.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:24.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:25.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:25.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:26.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:26.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:27.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:27.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:28.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:28.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:29.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:29.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:30.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:30.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:31.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:31.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:33.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:33.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:34.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:34.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:35.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:35.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:36.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:36.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:37.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:36.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:38.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:38.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:38.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:38.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:39.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:39.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:40.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:40.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:41.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:41.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:42.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:42.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:43.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:43.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:44.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:44.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:45.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:45.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:46.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:46.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:47.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:47.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:48.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:48.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:49.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:49.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:50.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:50.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:51.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:51.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:52.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:52.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:53.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:53.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:54.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:54.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:55.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:55.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:56.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:56.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:57.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:57.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:58.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:58.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:44:59.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:44:59.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:00.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:00.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:01.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:01.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:02.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:02.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:03.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:03.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:04.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:04.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:05.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:05.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:06.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:06.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:07.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:07.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:08.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:08.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:09.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:09.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:10.501 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:45:10.501 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:45:10.502 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:45:10.502 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:45:10.502 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:45:10.509 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:45:10.509 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T06:45:10.509 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T06:45:10.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:45:10.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:10.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:11.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:11.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:12.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:12.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:13.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:13.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:14.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:14.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:15.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:15.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:16.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:16.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:17.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:17.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:18.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:18.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:19.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:19.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:20.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:20.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:21.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:21.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:22.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:22.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:23.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:23.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:24.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:24.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:25.509 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T06:45:25.510 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T06:45:25.510 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:45:25.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:25.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:25.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:25.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:26.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:26.667 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:26.667 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:45:26.667 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:26.667 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:45:26.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:26.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:27.692 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1098,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:27.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:27.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:45:27.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:27.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:28.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1098,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:28.743 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:28.743 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:45:28.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:28.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:29.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1098,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:29.787 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:45:29.788 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:29.788 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:45:29.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:29.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:30.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:30.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:30.910 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:45:31.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:31.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:31.912 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:45:31.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:31.912 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:32.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:32.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:32.909 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:32.931 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:32.931 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:45:32.931 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:45:32.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:32.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:45:33.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:33.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:33.927 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1098,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:33.957 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:33.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:45:34.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:34.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:34.941 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1098,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:34.971 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:34.971 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:45:35.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:35.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:35.975 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1098,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:35.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T06:45:35.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:45:35.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T06:45:35.997 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:45:35.997 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T06:45:35.997 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:45:36.015 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T06:45:36.015 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:45:36.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:36.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:36.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:36.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:37.007 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:37.029 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:37.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:37.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:38.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:38.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:39.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:39.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:40.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:40.031 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:40.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:40.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:41.011 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:41.043 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:41.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:41.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:42.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:42.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:43.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:43.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:44.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:44.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:44.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:44.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:45.033 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:45.054 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1098,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:45.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:45.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:46.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:46.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:47.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:47.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:48.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:48.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:45:48.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:48.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:49.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:49.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:50.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:50.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:51.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:51.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:52.109 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1100,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:52.131 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1100,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:45:52.131 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:45:52.132 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:52.132 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:45:52.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:52.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:53.118 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1100,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:53.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:45:53.150 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:45:53.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:53.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:54.180 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1100,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:45:54.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:45:54.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:54.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:55.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:55.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:56.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:56.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:57.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:57.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:59.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:59.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:45:59.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:45:59.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:00.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:00.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:01.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:01.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:02.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:02.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:03.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:03.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:04.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:04.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:05.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:05.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:06.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:06.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:07.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:07.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:08.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:08.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:09.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:09.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:10.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:10.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:11.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:11.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:12.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:12.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:13.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:13.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:14.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:14.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:16.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:16.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:17.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:17.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:18.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:18.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:19.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:19.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:19.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:19.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:21.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:21.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:22.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:22.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:23.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:23.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:24.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:24.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:25.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:25.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:26.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:26.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:27.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:27.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:28.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:28.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:29.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:29.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:30.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:30.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:31.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:31.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:32.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:32.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:33.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:33.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:34.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:34.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:35.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:35.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:36.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:36.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:37.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:37.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:38.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:38.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:39.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:39.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:40.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:40.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:41.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:41.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:41.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:41.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:43.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:43.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:44.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:44.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:44.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:44.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:45.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:45.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:46.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:46.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:47.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:47.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:48.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:48.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:49.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:49.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:50.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:50.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:51.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:51.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:52.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:52.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:53.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:53.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:54.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:54.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:55.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:55.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:57.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:57.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:58.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:58.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:46:59.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:46:59.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:00.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:00.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:01.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:01.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:01.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:01.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:03.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:03.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:03.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:03.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:05.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:05.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:06.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:06.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:07.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:07.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:08.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:08.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:09.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:09.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:10.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:10.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:11.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:11.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:12.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:12.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:13.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:13.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:14.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:14.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:15.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:15.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:16.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:16.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:17.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:17.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:18.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:18.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:19.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:19.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:20.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:20.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:21.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:21.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:22.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:22.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:23.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:23.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:24.203 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T06:47:24.204 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T06:47:24.204 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T06:47:24.204 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T06:47:24.204 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T06:47:24.210 INFO:tasks.ceph.mon.b:Started 2019-10-01T06:47:24.210 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:47:24.211 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:24.211 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:24.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:24.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:25.191 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1100,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:25.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1100,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:25.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:25.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:26.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:26.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:27.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:27.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:28.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:28.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:28.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:28.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:29.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:29.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:30.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:30.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:31.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:31.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:31.404 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:31.435 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:31.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:47:31.435 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:31.436 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:47:32.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:32.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:32.442 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1104,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:32.464 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:32.464 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:47:33.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:33.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:33.424 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1104,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:33.455 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:33.456 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:47:34.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:34.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:34.463 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1104,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:34.484 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:47:34.485 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:34.485 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:47:35.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:35.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:35.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:47:36.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:36.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:36.620 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:47:36.621 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:36.621 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:37.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:37.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:37.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:37.668 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:37.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:47:37.669 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:47:37.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:37.669 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:47:38.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:38.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:38.625 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1104,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:38.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:38.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:47:39.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:39.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:39.613 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1104,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:39.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:39.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:47:40.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:40.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:40.639 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1104,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:40.662 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T06:47:40.663 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:47:40.663 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T06:47:40.663 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:47:40.664 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T06:47:40.664 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:47:40.694 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T06:47:40.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:47:40.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:40.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:41.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:41.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:41.767 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:41.799 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:42.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:42.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:43.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:43.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:44.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:44.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:44.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:44.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:45.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:45.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:45.879 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:45.909 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1104,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:46.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:46.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:47.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:47.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:48.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:48.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:48.911 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:48.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:47:49.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:49.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:50.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:50.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:51.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:51.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:52.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:52.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:53.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:53.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:54.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:54.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:54.712 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1108,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:54.744 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1108,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:47:54.744 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:47:54.744 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:54.744 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:47:55.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:55.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:55.753 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1108,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:55.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:47:55.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:47:56.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:56.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:56.762 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1108,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:47:56.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:47:57.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:57.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:58.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:58.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:47:59.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:47:59.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:00.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:00.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:01.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:01.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:02.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:02.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:03.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:03.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:04.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:04.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:05.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:05.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:06.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:06.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:07.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:07.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:08.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:08.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:09.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:09.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:10.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:10.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:11.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:11.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:12.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:12.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:13.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:13.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:14.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:14.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:15.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:15.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:16.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:16.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:17.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:17.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:18.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:18.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:19.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:19.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:20.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:20.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:21.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:21.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:22.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:22.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:23.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:23.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:24.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:24.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:25.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:25.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:26.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:26.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:27.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:27.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:28.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:28.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:29.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:29.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:30.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:30.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:31.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:31.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:32.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:32.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:33.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:33.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:34.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:34.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:35.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:35.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:36.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:36.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:37.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:37.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:38.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:38.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:39.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:39.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:40.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:40.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:41.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:41.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:42.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:42.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:43.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:43.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:44.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:44.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:45.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:45.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:46.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:46.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:47.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:47.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:48.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:48.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:49.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:49.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:50.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:50.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:51.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:51.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:52.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:52.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:53.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:53.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:54.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:54.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:55.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:55.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:56.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:56.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:57.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:57.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:58.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:58.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:48:59.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:48:59.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:00.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:00.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:01.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:01.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:02.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:02.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:03.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:03.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:04.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:04.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:05.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:05.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:06.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:06.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:07.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:07.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:08.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:08.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:09.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:09.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:10.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:10.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:11.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:11.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:12.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:12.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:13.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:13.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:14.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:14.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:15.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:15.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:16.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:16.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:17.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:17.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:18.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:18.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:19.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:19.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:20.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:20.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:21.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:21.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:22.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:22.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:23.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:23.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:24.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:24.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:25.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:25.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:26.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:26.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:26.786 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T06:49:26.787 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T06:49:26.787 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T06:49:26.787 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:26.787 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T06:49:26.793 INFO:tasks.ceph.mon.c:Started 2019-10-01T06:49:26.793 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:49:26.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:26.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:49:27.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:27.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:27.754 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1108,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:27.785 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1108,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:28.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:28.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:29.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:29.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:30.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:30.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:30.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:30.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:49:31.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:31.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:32.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:32.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:33.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:33.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:34.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:34.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:35.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:35.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:36.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:36.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:37.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:37.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:37.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1114,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:38.021 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1114,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:38.021 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:49:38.021 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:38.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:49:38.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:38.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:39.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1114,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:39.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:39.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:49:39.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:39.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:40.140 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1114,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:40.172 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:40.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:49:40.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:40.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:41.101 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1114,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:41.123 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:49:41.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:41.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:49:41.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:41.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:42.225 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:49:42.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:42.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:43.225 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:49:43.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:43.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:49:43.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:43.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:44.248 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1114,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:44.274 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1114,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:49:44.274 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:49:44.274 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:49:44.274 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:44.274 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:49:44.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:44.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:45.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:45.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:45.284 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1114,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:45.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:45.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:49:46.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:46.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:46.352 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1114,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:46.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:49:46.375 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:49:47.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:47.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:47.373 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1114,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:49:47.394 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:49:47.394 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T06:49:47.395 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:49:47.395 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:49:47.395 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:49:47.395 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:49:47.415 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:49:47.415 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:49:47.415 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:49:47.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:49:48.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:48.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:49.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:49.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:50.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:50.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:51.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:51.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:52.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:52.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:53.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:53.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:54.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:54.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:55.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:55.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:56.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:56.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:57.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:57.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:58.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:58.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:49:59.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:49:59.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:00.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:00.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:01.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:01.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:02.417 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:50:02.417 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:50:02.418 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:50:02.418 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:50:02.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:50:02.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:02.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:03.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:03.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:04.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:04.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:05.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:05.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:06.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:06.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:07.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:07.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:08.072 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1118,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:50:08.102 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1118,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:50:08.102 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:50:08.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:50:08.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:50:08.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:08.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:09.118 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1118,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:50:09.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:50:09.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:50:09.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:09.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:10.090 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1118,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:50:10.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:50:10.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:10.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:11.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:11.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:12.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:12.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:13.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:13.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:14.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:14.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:15.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:15.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:16.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:16.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:17.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:17.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:18.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:18.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:19.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:19.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:20.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:20.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:21.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:21.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:22.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:22.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:23.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:23.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:24.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:24.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:25.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:25.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:26.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:26.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:27.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:27.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:28.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:28.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:29.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:29.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:30.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:30.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:31.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:31.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:32.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:32.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:33.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:33.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:34.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:34.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:35.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:35.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:36.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:36.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:37.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:37.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:38.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:38.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:39.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:39.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:40.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:40.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:41.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:41.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:42.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:42.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:43.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:43.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:44.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:44.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:45.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:45.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:46.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:46.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:47.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:47.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:48.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:48.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:49.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:49.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:50.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:50.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:51.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:51.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:52.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:52.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:53.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:53.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:54.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:54.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:55.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:55.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:56.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:56.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:57.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:57.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:58.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:58.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:50:59.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:50:59.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:00.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:00.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:01.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:01.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:02.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:02.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:03.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:03.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:04.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:04.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:05.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:05.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:06.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:06.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:07.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:07.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:08.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:08.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:09.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:09.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:10.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:10.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:11.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:11.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:12.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:12.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:13.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:13.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:14.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:14.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:15.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:15.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:16.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:16.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:17.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:17.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:18.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:18.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:19.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:19.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:20.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:20.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:21.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:21.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:22.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:22.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:23.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:23.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:24.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:24.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:25.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:25.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:26.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:26.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:27.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:27.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:28.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:28.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:29.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:29.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:30.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:30.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:31.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:31.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:32.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:32.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:33.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:33.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:34.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:34.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:35.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:35.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:36.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:36.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:37.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:37.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:38.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:38.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:39.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:39.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:40.113 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:51:40.113 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:51:40.113 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:51:40.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:40.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:51:40.128 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:51:40.129 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T06:51:40.129 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T06:51:40.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T06:51:40.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:40.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:41.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:41.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:42.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:42.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:43.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:43.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:44.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:44.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:45.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:45.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:46.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:46.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:47.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:47.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:48.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:48.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:49.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:49.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:50.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:50.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:51.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:51.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:52.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:52.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:53.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:53.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:54.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:54.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:55.130 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T06:51:55.131 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T06:51:55.131 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:51:55.131 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:55.131 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:51:55.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:55.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:56.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:51:56.156 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:51:56.156 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:51:56.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:56.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:51:56.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:56.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:57.157 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1126,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:51:57.180 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:57.180 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:51:57.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:57.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:58.189 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1126,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:51:58.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:58.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:58.210 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:58.210 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:51:59.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:51:59.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:51:59.280 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1126,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:51:59.301 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:51:59.301 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:51:59.301 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:52:00.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:00.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:00.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:52:01.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:01.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:01.389 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:52:01.390 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:01.390 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:52:02.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:02.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:02.378 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:02.408 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:02.408 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:52:02.408 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:52:02.409 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:02.409 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:52:03.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:03.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:03.425 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1126,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:52:03.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:03.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:52:04.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:04.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:04.441 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1126,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:52:04.463 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:04.463 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:52:05.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:05.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:05.576 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1126,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:52:05.599 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:52:05.599 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:52:05.599 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:52:05.599 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:52:05.600 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:52:05.600 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:52:05.621 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:52:05.622 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:52:05.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:05.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:52:06.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:06.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:06.669 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:06.690 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:07.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:07.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:08.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:08.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:09.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:09.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:09.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:09.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:52:10.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:10.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:10.684 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:10.707 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:11.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:11.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:12.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:12.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:13.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:13.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:13.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:13.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:52:14.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:14.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:14.707 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:14.729 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1126,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:15.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:15.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:16.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:16.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:17.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:17.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:17.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:17.730 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:52:18.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:18.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:19.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:19.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:20.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:20.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:21.096 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1130,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:21.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:21.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:21.118 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1130,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:52:21.118 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:52:21.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:21.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:52:22.119 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1130,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:52:22.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:22.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:22.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:52:22.142 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:52:23.130 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1130,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:52:23.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:52:23.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:23.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:24.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:24.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:25.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:25.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:26.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:26.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:27.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:27.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:28.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:28.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:29.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:29.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:30.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:30.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:31.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:31.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:32.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:32.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:33.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:33.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:34.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:34.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:35.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:35.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:36.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:36.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:37.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:37.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:38.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:38.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:39.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:39.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:40.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:40.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:41.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:41.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:42.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:42.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:43.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:43.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:44.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:44.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:45.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:45.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:46.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:46.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:47.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:47.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:48.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:48.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:49.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:49.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:50.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:50.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:51.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:51.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:52.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:52.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:53.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:53.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:54.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:54.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:55.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:55.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:56.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:56.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:57.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:57.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:58.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:58.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:52:59.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:52:59.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:00.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:00.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:01.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:01.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:02.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:02.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:03.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:03.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:04.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:04.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:05.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:05.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:06.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:06.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:07.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:07.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:08.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:08.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:09.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:09.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:10.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:10.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:11.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:11.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:12.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:12.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:13.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:13.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:14.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:14.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:15.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:15.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:16.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:16.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:17.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:17.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:18.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:18.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:19.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:19.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:20.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:20.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:21.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:21.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:22.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:22.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:23.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:23.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:24.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:24.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:25.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:25.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:26.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:26.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:27.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:27.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:28.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:28.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:29.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:29.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:30.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:30.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:31.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:31.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:32.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:32.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:33.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:33.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:34.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:34.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:35.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:35.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:36.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:36.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:37.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:37.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:38.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:38.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:39.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:39.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:40.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:40.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:41.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:41.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:42.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:42.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:43.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:43.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:44.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:44.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:45.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:45.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:46.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:46.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:47.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:47.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:48.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:48.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:49.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:49.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:50.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:50.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:51.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:51.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:52.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:52.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:53.153 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:53:53.153 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:53:53.153 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:53:53.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:53:53.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:53:53.160 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:53:53.160 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:53:53.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:53:53.161 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:53:53.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:53.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:54.118 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1130,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:53:54.140 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1130,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:53:54.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:54.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:55.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:55.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:56.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:56.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:57.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:53:57.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:53:57.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:57.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:58.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:58.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:53:59.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:53:59.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:00.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:00.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:00.983 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:01.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:01.014 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:54:01.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:01.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:54:01.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:01.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:01.968 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1138,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:01.991 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:01.991 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:54:02.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:02.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:02.983 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1138,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:03.006 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:03.006 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:54:03.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:03.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:03.995 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1138,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:04.017 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:54:04.017 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:04.017 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:54:04.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:04.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:05.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:54:05.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:05.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:06.119 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:54:06.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:06.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:54:06.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:06.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:07.130 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:07.162 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:07.162 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:54:07.163 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:54:07.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:07.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:54:07.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:07.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:08.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1138,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:08.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:08.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:54:08.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:08.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:09.062 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1138,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:09.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:09.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:54:09.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:09.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:10.049 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1138,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:10.079 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:54:10.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:54:10.144 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:54:10.144 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:54:10.145 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:54:10.145 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:54:10.161 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:54:10.162 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:54:10.162 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:10.162 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:54:10.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:10.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:11.172 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:11.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:11.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:11.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:12.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:12.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:13.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:13.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:14.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:14.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:54:14.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:14.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:15.203 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:15.234 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:15.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:15.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:16.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:16.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:17.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:17.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:18.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:18.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:54:18.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:18.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:19.231 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:19.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1138,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:19.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:19.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:20.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:20.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:21.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:21.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:22.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:22.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:54:22.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:22.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:23.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:23.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:24.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:24.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:25.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:25.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:25.553 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:25.583 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:54:25.583 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:54:25.584 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:25.584 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:54:26.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:26.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:26.653 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1142,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:26.683 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:54:26.683 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:54:27.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:27.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:27.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1142,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:54:27.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:54:28.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:28.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:29.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:29.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:30.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:30.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:31.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:31.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:32.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:32.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:33.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:33.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:34.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:34.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:35.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:35.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:36.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:36.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:37.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:37.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:38.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:38.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:39.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:39.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:40.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:40.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:41.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:41.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:42.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:42.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:43.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:43.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:44.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:44.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:45.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:45.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:46.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:46.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:47.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:47.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:48.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:48.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:49.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:49.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:50.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:50.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:51.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:51.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:52.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:52.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:53.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:53.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:54.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:54.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:55.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:55.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:56.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:56.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:57.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:57.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:58.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:58.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:54:59.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:54:59.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:00.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:00.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:01.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:01.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:02.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:02.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:03.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:03.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:04.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:04.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:05.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:05.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:06.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:06.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:07.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:07.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:08.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:08.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:09.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:09.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:10.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:10.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:11.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:11.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:12.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:12.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:13.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:13.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:14.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:14.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:15.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:15.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:16.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:16.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:17.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:17.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:18.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:18.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:19.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:19.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:20.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:20.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:21.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:21.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:22.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:22.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:23.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:23.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:24.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:24.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:25.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:25.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:26.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:26.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:27.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:27.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:28.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:28.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:29.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:29.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:30.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:30.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 75 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:31.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:31.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:32.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:32.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:33.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:33.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:34.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:34.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:35.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:35.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:36.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:36.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:37.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:37.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:38.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:38.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:39.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:39.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:40.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:40.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:41.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:41.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:42.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:42.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:43.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:43.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:44.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:44.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:45.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:45.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:46.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:46.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:47.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:47.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:48.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:48.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:49.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:49.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:50.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:50.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:51.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:51.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:52.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:52.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:53.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:53.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:54.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:54.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:55.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:55.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:56.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:56.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:57.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:57.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:57.716 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:55:57.716 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:55:57.716 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:55:57.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:55:57.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:55:57.722 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:55:57.722 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:55:57.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:55:57.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:55:58.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:58.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:55:58.653 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:55:58.676 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1142,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:55:59.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:55:59.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:00.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:00.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:01.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:01.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:01.678 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:01.678 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:02.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:02.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:02.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:02.667 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:02.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:56:02.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:02.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:56:03.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:03.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:03.662 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1148,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:03.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:03.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:56:04.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:04.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:04.693 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1148,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:04.714 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:04.714 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:56:05.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:05.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:05.728 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1148,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:05.749 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:56:05.750 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:05.750 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:56:06.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:06.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:06.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:56:07.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:07.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:07.845 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:56:07.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:07.846 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:08.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:08.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:08.828 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:08.846 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:08.847 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:56:08.847 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:56:08.847 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:08.847 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:56:09.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:09.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:09.832 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1148,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:09.863 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:09.864 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:56:10.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:10.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:10.865 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1148,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:10.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:10.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:56:11.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:11.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:11.874 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1148,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:11.896 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:56:11.896 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:56:11.897 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:56:11.897 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:56:11.897 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:56:11.897 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:56:11.913 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:56:11.914 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:56:11.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:11.914 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:12.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:12.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:12.976 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:13.005 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:13.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:13.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:14.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:14.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:15.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:15.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:16.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:16.007 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:16.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:16.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:16.992 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:17.024 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:17.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:17.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:18.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:18.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:19.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:19.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:20.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:20.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:20.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:20.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:21.001 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:21.023 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1148,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:21.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:21.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:22.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:22.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:23.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:23.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:24.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:24.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:56:24.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:24.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:25.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:25.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:26.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:26.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:27.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:27.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:27.380 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1152,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:27.412 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1152,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:56:27.412 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:56:27.412 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:27.413 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:56:28.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:28.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:28.391 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1152,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:28.415 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:56:28.416 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:56:29.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:29.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:29.410 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1152,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:56:29.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:56:30.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:30.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:31.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:31.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:32.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:32.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:33.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:33.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:34.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:34.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:35.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:35.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:36.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:36.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:37.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:37.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:38.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:38.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:39.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:39.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:40.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:40.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:41.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:41.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:42.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:42.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:43.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:43.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:44.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:44.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:45.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:45.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:46.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:46.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:47.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:47.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:48.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:48.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:49.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:49.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:50.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:50.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:51.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:51.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:52.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:52.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:53.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:53.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:54.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:54.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:55.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:55.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:56.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:56.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:57.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:57.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:58.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:58.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:56:59.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:56:59.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:00.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:00.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:01.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:01.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:02.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:02.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:03.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:03.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:04.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:04.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:05.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:05.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:06.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:06.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:07.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:07.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:08.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:08.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:09.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:09.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:10.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:10.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:11.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:11.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:12.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:12.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:13.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:13.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:14.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:14.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:15.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:15.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:16.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:16.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:17.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:17.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:18.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:18.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:19.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:19.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:20.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:20.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:21.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:21.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:22.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:22.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:23.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:23.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:24.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:24.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:25.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:25.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:26.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:26.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:27.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:27.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:28.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:28.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:29.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:29.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:30.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:30.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:31.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:31.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:32.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:32.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:33.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:33.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:34.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:34.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:35.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:35.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:36.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:36.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:37.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:37.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:38.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:38.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:39.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:39.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:40.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:40.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:41.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:41.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:42.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:42.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:43.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:43.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:44.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:44.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:45.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:45.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:46.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:46.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:47.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:47.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:48.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:48.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:49.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:49.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:50.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:50.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:51.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:51.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:52.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:52.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:53.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:53.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:54.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:54.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:55.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:55.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:56.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:56.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:57.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:57.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:58.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:58.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:57:59.432 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:57:59.432 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T06:57:59.432 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T06:57:59.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:57:59.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T06:57:59.438 INFO:tasks.ceph.mon.a:Started 2019-10-01T06:57:59.438 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:57:59.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:57:59.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:57:59.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:57:59.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:00.406 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1152,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:00.437 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1152,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:00.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:00.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:01.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:01.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:02.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:02.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:03.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:03.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:03.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:03.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:04.436 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:04.458 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:04.458 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:58:04.458 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:04.458 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:58:04.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:04.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:05.460 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1158,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:05.481 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:05.482 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:58:05.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:05.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:06.501 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1158,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:06.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:06.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:58:06.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:06.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:07.543 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1158,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:07.563 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T06:58:07.564 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:07.564 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T06:58:07.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:07.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:08.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T06:58:08.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:08.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:09.676 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T06:58:09.677 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:09.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:09.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:09.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:10.744 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:10.765 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:10.766 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T06:58:10.766 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T06:58:10.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:10.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T06:58:10.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:10.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:11.711 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1158,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:11.733 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:11.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:58:11.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:11.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:12.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1158,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:12.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:12.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:58:12.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:12.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:13.784 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1158,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:13.807 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T06:58:13.807 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T06:58:13.807 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T06:58:13.808 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T06:58:13.808 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T06:58:13.808 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T06:58:13.827 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T06:58:13.828 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T06:58:13.828 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:13.828 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:13.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:13.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:14.873 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:14.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:14.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:14.905 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:15.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:15.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:16.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:16.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:17.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:17.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:17.907 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:17.907 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:18.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:18.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:18.900 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:18.923 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:19.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:19.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:20.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:20.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:21.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:21.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:21.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:21.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:22.896 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:22.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:22.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:22.917 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1158,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:23.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:23.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:24.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:24.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:25.917 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:25.918 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T06:58:25.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:25.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:26.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:26.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:27.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:27.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:28.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:28.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:29.024 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:29.046 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T06:58:29.046 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T06:58:29.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:29.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T06:58:29.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:29.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:30.083 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1162,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:30.113 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T06:58:30.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T06:58:30.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:30.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:31.061 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1162,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T06:58:31.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T06:58:31.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:31.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:32.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:32.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:33.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:33.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:34.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:34.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:35.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:35.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:36.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:36.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:37.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:37.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:38.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:38.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:39.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:39.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:40.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:40.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:41.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:41.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:42.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:42.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:43.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:43.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:44.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:44.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:45.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:45.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:46.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:46.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:47.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:47.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:48.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:48.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:49.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:49.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:50.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:50.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:51.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:51.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:52.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:52.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:54.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:54.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:54.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:54.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:56.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:56.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:57.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:57.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:58.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:58.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:58:59.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:58:59.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:00.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:00.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:01.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:01.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:02.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:02.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:03.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:03.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:04.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:04.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:05.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:05.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:06.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:06.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:07.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:07.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:08.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:08.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:09.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:09.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:10.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:10.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:11.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:11.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:12.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:12.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:13.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:13.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:14.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:14.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:15.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:15.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:16.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:16.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:17.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:17.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:18.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:18.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:19.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:19.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:20.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:20.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:21.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:21.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:22.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:22.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:23.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:23.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:24.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:24.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:25.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:25.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:26.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:26.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:27.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:27.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:28.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:28.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:29.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:29.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:30.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:30.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:31.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:31.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:32.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:32.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:33.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:33.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:34.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:34.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:35.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:35.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:36.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:36.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:37.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:37.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:38.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:38.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:39.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:39.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:40.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:40.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:41.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:41.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:42.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:42.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:43.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:43.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:44.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:44.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:45.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:45.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:46.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:46.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:47.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:47.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:48.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:48.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:49.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:49.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:50.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:50.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:51.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:51.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:52.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:52.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:53.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:53.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:54.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:54.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:55.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:55.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:56.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:56.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:57.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:57.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:58.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:58.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T06:59:58.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T06:59:58.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:00.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:00.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:01.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:01.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:01.085 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:00:01.085 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:00:01.085 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:00:01.085 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:01.085 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:00:01.091 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:00:01.091 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:00:01.091 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:01.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:01.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:01.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:02.086 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:02.115 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1162,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:03.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:03.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:04.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:04.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:05.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:05.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:05.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:05.117 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:06.101 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:06.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:06.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:06.125 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:06.125 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:00:06.125 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:06.125 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:00:06.730 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T07:00:06.728+0000 7f18359cd700 0 --2- 172.21.6.124:0/3640115544 >> v2:172.21.6.124:3300/0 conn(0x7f183007b830 0x7f183007eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T07:00:07.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:07.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:07.363 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1168,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:07.395 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:07.395 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:00:08.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:08.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:08.401 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1168,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:08.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:08.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:00:09.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:09.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:09.430 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1168,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:09.452 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:00:09.452 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:09.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:00:10.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:10.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:10.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:00:11.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:11.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:11.595 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:00:11.595 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:11.595 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:12.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:12.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:12.621 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:12.642 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:12.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:00:12.643 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:00:12.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:12.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:00:13.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:13.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:13.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1168,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:13.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:13.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:00:14.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:14.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:14.685 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1168,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:14.704 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:14.705 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:00:15.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:15.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:15.702 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1168,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:15.733 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:00:15.733 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:00:15.733 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:00:15.733 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:00:15.733 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:00:15.734 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:00:15.767 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:00:15.767 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:00:15.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:15.768 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:16.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:16.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:16.779 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:16.801 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:17.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:17.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:18.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:18.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:19.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:19.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:19.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:19.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:20.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:20.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:20.772 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:20.794 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:21.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:21.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:22.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:22.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:23.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:23.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:23.794 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:23.795 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:24.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:24.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:24.798 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:24.827 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1168,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:25.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:25.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:26.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:26.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:27.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:27.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:27.829 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:27.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:00:28.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:28.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:29.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:29.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:30.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:30.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:31.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:31.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:32.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:32.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:33.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:33.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:34.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:34.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:35.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:35.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:36.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:36.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:37.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:37.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:38.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:38.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:38.852 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:38.883 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:00:38.884 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:00:38.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:38.884 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:00:39.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:39.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:39.867 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1172,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:00:39.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:00:39.889 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:00:40.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:40.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:40.900 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1172,"quorum":[1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:00:40.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:00:41.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:41.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:42.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:42.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:43.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:43.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:44.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:44.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:45.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:45.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:46.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:46.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:47.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:47.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:48.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:48.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:49.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:49.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:50.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:50.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:51.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:51.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:52.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:52.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:53.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:53.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:54.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:54.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:55.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:55.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:56.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:56.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:57.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:57.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:58.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:58.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:00:59.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:00:59.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:00.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:00.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:01.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:01.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:02.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:02.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:03.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:03.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:04.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:04.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:05.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:05.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:06.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:06.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:07.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:07.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:08.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:08.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:09.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:09.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:10.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:10.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:11.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:11.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:12.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:12.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:13.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:13.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:14.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:14.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:15.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:15.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:16.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:16.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:17.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:17.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:18.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:18.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:19.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:19.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:20.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:20.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:21.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:21.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:22.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:22.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:23.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:23.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:24.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:24.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:25.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:25.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:26.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:26.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:27.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:27.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:28.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:28.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:29.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:29.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:30.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:30.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:31.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:31.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:32.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:32.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:33.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:33.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:34.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:34.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:35.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:35.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:36.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:36.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:37.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:37.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:38.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:38.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:39.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:39.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:40.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:40.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:41.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:41.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:42.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:42.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:43.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:43.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:44.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:44.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:45.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:45.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:46.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:46.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:47.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:47.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:48.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:48.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:49.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:49.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:50.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:50.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:51.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:51.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:52.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:52.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:53.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:53.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:54.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:54.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:55.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:55.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:56.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:56.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:57.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:57.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:58.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:58.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:01:59.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:01:59.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:00.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:00.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:01.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:01.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:02.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:02.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:03.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:03.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:04.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:04.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:05.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:05.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:06.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:06.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:07.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:07.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:08.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:08.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:09.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:09.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:10.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:10.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:10.925 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:02:10.926 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:02:10.926 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:02:10.926 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:02:10.926 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:02:10.934 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:02:10.934 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:02:10.934 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:10.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:11.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:11.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:11.944 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":96,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:11.975 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":96,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:12.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:12.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:13.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:13.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:14.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:14.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:14.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:14.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:15.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:15.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:15.964 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":100,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:15.985 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":100,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:16.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:16.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:17.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:17.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:18.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:18.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:18.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:18.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:19.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:19.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:20.003 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":104,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:20.025 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":104,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:20.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:20.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:21.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:21.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:22.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:22.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:23.025 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:23.025 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:23.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:23.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:24.012 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":108,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:24.034 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":108,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:24.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:24.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:25.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:25.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:26.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:26.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:27.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:27.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:27.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:27.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:28.046 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":112,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:28.077 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":112,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:28.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:28.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:29.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:29.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:30.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:30.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:31.078 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:31.078 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:31.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:31.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:32.090 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":116,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:32.113 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":116,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:32.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:32.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:33.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:33.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:34.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:34.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:35.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:35.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:35.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:35.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:36.120 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":120,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:36.143 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1172,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":120,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:36.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:36.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:37.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:37.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:38.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:38.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:39.144 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:39.145 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:39.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:39.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:40.160 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:40.182 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:40.183 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:02:40.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:40.183 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:02:40.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:40.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:41.183 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1176,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:02:41.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:41.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:02:41.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:41.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:42.197 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1176,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:02:42.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:42.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:02:42.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:42.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:43.182 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1176,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:02:43.204 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:02:43.205 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:43.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:02:43.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:43.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:44.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:02:44.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:44.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:45.350 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:02:45.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:45.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:45.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:45.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:46.323 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:46.345 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:46.345 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:02:46.346 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:02:46.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:46.346 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:02:46.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:46.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:47.325 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1176,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:02:47.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:47.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:02:47.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:47.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:48.360 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1176,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:02:48.380 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:48.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:02:48.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:48.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:49.393 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1176,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:02:49.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T07:02:49.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:02:49.416 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T07:02:49.416 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:02:49.416 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T07:02:49.417 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:02:49.436 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T07:02:49.437 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:02:49.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:49.437 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:49.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:49.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:50.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:50.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:50.495 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:50.518 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:51.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:51.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:52.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:52.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:53.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:53.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:53.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:53.519 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:54.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:54.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:54.504 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:54.526 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1176,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:02:55.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:55.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:56.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:56.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:57.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:57.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:57.528 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:02:57.528 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:02:58.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:58.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:02:59.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:02:59.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:00.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:00.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:01.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:01.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:02.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:02.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:03.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:03.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:04.488 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:03:04.512 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:03:04.512 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:03:04.513 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:03:04.513 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:03:04.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:04.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:05.527 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1180,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:03:05.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:03:05.558 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:03:05.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:05.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:06.561 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1180,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:03:06.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:03:06.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:06.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:07.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:07.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:08.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:08.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:09.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:09.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:10.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:10.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:11.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:11.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:12.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:12.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:13.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:13.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:14.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:14.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:15.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:15.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:16.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:16.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:17.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:17.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:18.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:18.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:19.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:19.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:20.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:20.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:21.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:21.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:22.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:22.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:23.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:23.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:24.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:24.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:25.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:25.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:26.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:26.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:27.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:27.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:28.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:28.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:29.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:29.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:30.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:30.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:31.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:31.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:32.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:32.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:33.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:33.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:34.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:34.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:35.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:35.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:36.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:36.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:37.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:37.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:38.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:38.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:39.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:39.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:40.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:40.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:41.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:41.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:42.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:42.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:43.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:43.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:44.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:44.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:45.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:45.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:46.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:46.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:47.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:47.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:48.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:48.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:49.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:49.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:50.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:50.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:51.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:51.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:52.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:52.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:53.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:53.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:54.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:54.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:55.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:55.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:56.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:56.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:57.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:57.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:58.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:58.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:03:59.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:03:59.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:00.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:00.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:01.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:01.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:02.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:02.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:03.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:03.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:04.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:04.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:05.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:05.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:06.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:06.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:07.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:07.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:08.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:08.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:09.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:09.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:10.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:10.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:11.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:11.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:12.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:12.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:13.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:13.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:14.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:14.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:15.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:15.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:16.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:16.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:17.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:17.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:18.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:18.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:19.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:19.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:20.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:20.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:21.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:21.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:22.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:22.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:23.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:23.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:24.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:24.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:25.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:25.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:26.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:26.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:27.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:27.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:28.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:28.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:29.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:29.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:30.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:30.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:31.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:31.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:32.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:32.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:33.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:33.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:34.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:34.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:35.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:35.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:36.584 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:04:36.584 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:04:36.585 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:04:36.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:36.585 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:04:36.590 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:04:36.591 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:04:36.591 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:36.591 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:36.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:36.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:37.571 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:37.602 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:37.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:37.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:38.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:38.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:39.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:39.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:40.603 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:40.603 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:40.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:40.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:41.590 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:41.622 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:41.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:41.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:42.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:42.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:43.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:43.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:44.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:44.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:44.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:44.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:45.601 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:45.633 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1180,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:45.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:45.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:46.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:46.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:47.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:47.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:48.634 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:48.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:48.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:48.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:49.635 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:49.667 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:49.667 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:04:49.667 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:49.667 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:04:49.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:49.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:50.668 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1182,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:04:50.691 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:50.691 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:04:50.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:50.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:51.698 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1182,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:04:51.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:51.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:04:51.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:51.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:52.688 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1182,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:04:52.710 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:04:52.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:52.711 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:04:52.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:52.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:53.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:04:53.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:53.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:54.839 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:04:54.840 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:54.840 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:54.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:54.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:55.832 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:55.853 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:55.854 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:04:55.854 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:04:55.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:55.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:04:55.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:55.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:56.811 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1182,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:04:56.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:56.842 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:04:56.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:56.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:57.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:57.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:57.897 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1182,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:04:57.926 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:57.927 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:04:58.860 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1182,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:04:58.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:58.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:58.886 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:04:58.886 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:04:58.886 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:04:58.886 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:04:58.886 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:04:58.887 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:04:58.909 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:04:58.910 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:04:58.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:04:58.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:04:59.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:04:59.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:04:59.922 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:04:59.954 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:00.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:00.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:01.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:01.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:02.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:02.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:02.954 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:05:02.955 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:05:03.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:03.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:03.928 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:03.955 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:04.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:04.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:05.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:05.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:06.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:06.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:06.956 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:05:06.956 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:05:07.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:07.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:07.975 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:08.004 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1182,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:08.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:08.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:09.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:09.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:10.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:10.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:11.006 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:05:11.006 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:05:11.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:11.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:12.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:12.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:13.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:13.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:14.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:14.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:15.058 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:15.077 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:05:15.077 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:05:15.077 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:05:15.078 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:05:15.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:15.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:16.050 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1184,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:05:16.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:05:16.073 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:05:16.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:16.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:17.003 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1184,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:05:17.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:05:17.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:17.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:18.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:18.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:19.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:19.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:20.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:20.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:21.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:21.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:22.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:22.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:23.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:23.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:24.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:24.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:25.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:25.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:26.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:26.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:27.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:27.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:28.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:28.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:29.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:29.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:30.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:30.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:31.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:31.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:32.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:32.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:33.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:33.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:34.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:34.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:35.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:35.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:36.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:36.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:37.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:37.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:38.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:38.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:39.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:39.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:40.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:40.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:41.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:41.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:42.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:42.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:43.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:43.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:44.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:44.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:45.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:45.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:46.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:46.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:47.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:47.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:48.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:48.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:49.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:49.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:50.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:50.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:51.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:51.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:52.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:52.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:53.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:53.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:54.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:54.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:55.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:55.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:56.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:56.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:57.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:57.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:58.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:58.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:05:59.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:05:59.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:00.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:00.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:01.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:01.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:02.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:02.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:03.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:03.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:05.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:05.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:06.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:06.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:07.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:07.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:08.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:08.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:09.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:09.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:10.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:10.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:11.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:11.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 81 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:12.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:12.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:13.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:13.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:14.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:14.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:15.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:15.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:16.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:16.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:17.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:17.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:18.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:18.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:19.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:19.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:19.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:19.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:20.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:20.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:21.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:21.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:22.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:22.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:23.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:23.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:24.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:24.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:25.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:25.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:26.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:26.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:27.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:27.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:28.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:28.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:29.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:29.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:30.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:30.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:31.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:31.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:32.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:32.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:33.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:33.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:34.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:34.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:35.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:35.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:36.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:36.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:37.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:37.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:38.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:38.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:39.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:39.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:41.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:41.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:42.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:42.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:43.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:43.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:44.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:44.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:45.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:45.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:45.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:45.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:47.027 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:06:47.027 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:06:47.027 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:06:47.027 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:06:47.027 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:06:47.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:47.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:47.036 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:06:47.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:06:47.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:06:47.037 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:06:48.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:48.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:48.051 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:48.081 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:49.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:49.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:50.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:50.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:51.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:51.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:51.082 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:06:51.083 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:06:52.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:52.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:52.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:52.100 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:53.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:53.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:54.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:54.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:55.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:55.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:55.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:06:55.101 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:06:56.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:56.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:56.088 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:56.115 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:06:56.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:56.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:57.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:57.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:58.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:58.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:06:59.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:06:59.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:06:59.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:06:59.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:00.140 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:00.164 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1184,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:00.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:00.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:01.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:01.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:02.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:02.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:03.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:03.165 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:03.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:03.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:04.167 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:04.197 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:04.197 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:07:04.197 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:04.197 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:07:04.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:04.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:05.185 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1188,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:05.210 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:05.210 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:07:05.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:05.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:06.164 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1188,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:06.186 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:06.186 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:07:06.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:06.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:07.165 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1188,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:07.196 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:07:07.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:07.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:07:07.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:07.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:08.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:07:08.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:08.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:09.335 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:07:09.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:09.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:09.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:09.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:10.329 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:10.350 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:10.351 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:07:10.351 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:07:10.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:10.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:07:10.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:10.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:11.337 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1188,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:11.359 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:11.359 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:07:11.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:11.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:12.348 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1188,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:12.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:12.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:07:12.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:12.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:13.365 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1188,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:13.386 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:07:13.387 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:07:13.387 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:07:13.387 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:07:13.387 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:07:13.387 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:07:13.407 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:07:13.407 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:07:13.407 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:13.407 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:13.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:13.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:14.395 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:14.425 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:14.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:14.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:15.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:15.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:16.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:16.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:17.426 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:17.426 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:17.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:17.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:18.426 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:18.451 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:18.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:18.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:19.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:19.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:20.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:20.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:21.452 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:21.452 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:21.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:21.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:22.457 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:22.479 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1188,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:22.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:22.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:23.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:23.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:24.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:24.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:25.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:25.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:07:25.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:25.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:26.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:26.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:27.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:27.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:28.180 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:28.203 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:07:28.203 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:07:28.203 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:28.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:07:28.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:28.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:29.202 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1190,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:07:29.225 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:07:29.225 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:07:29.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:29.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:30.232 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1190,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:07:30.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:07:30.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:30.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:31.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:31.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:32.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:32.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:33.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:33.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:34.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:34.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:35.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:35.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:36.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:36.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:37.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:37.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:38.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:38.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:39.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:39.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:40.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:40.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:41.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:41.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:43.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:43.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:44.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:44.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:45.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:45.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:46.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:46.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:47.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:47.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:48.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:48.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:49.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:49.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:50.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:50.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:51.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:51.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:52.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:52.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:53.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:53.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:54.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:54.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:55.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:55.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:56.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:56.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:57.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:57.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:58.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:58.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:07:59.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:07:59.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:00.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:00.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:01.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:01.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:02.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:02.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:03.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:03.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:04.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:04.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:05.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:05.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:06.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:06.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:07.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:07.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:08.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:08.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:09.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:09.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:10.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:10.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:11.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:11.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:12.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:12.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:13.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:13.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:14.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:14.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:15.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:15.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:16.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:16.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:17.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:17.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:18.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:18.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:19.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:19.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:20.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:20.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:21.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:21.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:22.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:22.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:23.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:23.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:24.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:24.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:25.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:25.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:26.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:26.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:27.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:27.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:28.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:28.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:29.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:29.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:30.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:30.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:31.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:31.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:32.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:32.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:33.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:33.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:34.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:34.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:35.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:35.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:36.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:36.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:37.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:37.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:38.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:38.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:39.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:39.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:40.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:40.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:41.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:41.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:42.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:42.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:43.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:43.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:44.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:44.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:45.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:45.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:46.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:46.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:47.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:47.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:48.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:48.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:49.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:49.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:50.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:50.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:51.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:51.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:52.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:52.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:53.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:53.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:54.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:54.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:55.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:55.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:56.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:56.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:57.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:57.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:58.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:58.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:08:59.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:08:59.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:00.250 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:09:00.250 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:09:00.251 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:09:00.251 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:09:00.251 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:09:00.280 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:09:00.281 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:09:00.281 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:00.281 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:00.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:00.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:01.290 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:01.311 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:01.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:01.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:02.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:02.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:03.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:03.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:04.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:04.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:04.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:04.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:05.310 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:05.338 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:05.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:05.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:06.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:06.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:07.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:07.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:08.340 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:08.340 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:08.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:08.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:09.316 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:09.346 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:09.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:09.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:10.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:10.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:11.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:11.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:12.347 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:12.347 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:12.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:12.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:13.340 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:13.363 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1190,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:13.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:13.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:14.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:14.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:15.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:15.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:16.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:16.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:16.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:16.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:17.332 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:17.354 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:17.354 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:09:17.355 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:17.355 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:09:17.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:17.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:18.354 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1194,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:09:18.385 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:18.385 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:09:18.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:18.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:19.366 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1194,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:09:19.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:19.396 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:09:19.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:19.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:20.425 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1194,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:09:20.453 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:09:20.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:20.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:09:20.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:20.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:21.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:09:21.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:21.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:22.568 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:09:22.569 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:22.569 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:22.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:22.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:23.552 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:23.582 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:23.582 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:09:23.582 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:09:23.583 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:23.583 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:09:23.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:23.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:24.582 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1194,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:09:24.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:24.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:09:24.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:24.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:25.595 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1194,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:09:25.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:25.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:09:26.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:25.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:26.637 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1194,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:09:26.659 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:09:26.659 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:09:26.659 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:09:26.659 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:09:26.659 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:09:26.660 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:09:26.709 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:09:26.710 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:09:26.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:26.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:26.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:26.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:27.717 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:27.740 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:27.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:27.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:28.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:28.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:29.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:29.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:30.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:30.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:30.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:30.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:31.760 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:31.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:31.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:31.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:32.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:32.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:33.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:33.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:34.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:34.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:34.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:34.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:35.778 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:35.808 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:35.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:35.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:36.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:36.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:37.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:37.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:38.810 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:38.810 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:09:38.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:38.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:39.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:39.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:40.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:40.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:41.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:41.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:41.991 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1198,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:42.023 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1198,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:09:42.023 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:09:42.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:42.023 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:09:42.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:42.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:43.031 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1198,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:09:43.053 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:09:43.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:09:43.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:43.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:44.023 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1198,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:09:44.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:09:44.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:44.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:45.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:45.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:46.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:46.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:47.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:47.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:48.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:48.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:49.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:49.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:50.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:50.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:51.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:51.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:52.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:52.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:53.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:53.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:54.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:54.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:55.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:55.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:56.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:56.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:57.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:57.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:58.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:58.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:09:59.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:09:59.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:00.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:00.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:01.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:01.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:02.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:02.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:03.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:03.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:04.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:04.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:05.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:05.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:06.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:06.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:07.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:07.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:08.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:08.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:09.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:09.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:10.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:10.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:11.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:11.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:12.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:12.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:13.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:13.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:14.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:14.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:15.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:15.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:16.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:16.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:17.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:17.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:18.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:18.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:19.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:19.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:20.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:20.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:21.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:21.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:22.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:22.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:23.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:23.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:24.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:24.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:25.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:25.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:26.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:26.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:27.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:27.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:28.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:28.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:29.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:29.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:30.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:30.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:31.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:31.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:32.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:32.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:33.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:33.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:34.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:34.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:35.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:35.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:36.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:36.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:37.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:37.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:38.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:38.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:39.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:39.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:40.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:40.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:41.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:41.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:42.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:42.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:43.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:43.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:44.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:44.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:45.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:45.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:46.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:46.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:47.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:47.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:48.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:48.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:49.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:49.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:50.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:50.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:51.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:51.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:52.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:52.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:53.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:53.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:54.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:54.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:55.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:55.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:57.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:57.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:58.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:57.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:10:58.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:10:58.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:00.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:00.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:01.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:01.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:02.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:02.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:03.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:03.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:03.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:03.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:04.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:04.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:05.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:05.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:06.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:06.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:07.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:07.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:08.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:08.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:09.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:09.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:10.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:10.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:11.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:11.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:12.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:12.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:13.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:13.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:14.053 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:11:14.054 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:11:14.054 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:11:14.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:14.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:11:14.061 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:11:14.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:11:14.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:14.061 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:14.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:14.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:15.086 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1198,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:15.110 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1198,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:15.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:15.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:16.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:16.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:17.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:17.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:18.111 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:18.112 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:18.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:18.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:19.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:19.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:20.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:20.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:21.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:21.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:22.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:22.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:22.973 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:23.001 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:23.001 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:11:23.002 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:23.002 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:11:23.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:23.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:23.997 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1204,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:11:24.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:24.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:11:24.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:24.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:25.447 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1204,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:11:25.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:25.469 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:11:25.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:25.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:26.450 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1204,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:11:26.472 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:11:26.472 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:26.472 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:11:26.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:26.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:27.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:11:27.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:27.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:28.590 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:11:28.591 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:28.591 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:28.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:28.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:29.600 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:29.621 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:29.621 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:11:29.621 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:11:29.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:29.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:11:29.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:29.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:30.625 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1204,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:11:30.650 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:30.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:11:30.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:30.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:31.628 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1204,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:11:31.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:31.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:11:31.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:31.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:32.567 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1204,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:11:32.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:11:32.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:11:32.589 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:11:32.589 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:11:32.589 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:11:32.590 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:11:32.611 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:11:32.611 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:11:32.611 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:32.611 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:33.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:33.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:33.605 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:33.627 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:33.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:33.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:34.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:34.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:36.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:36.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:36.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:36.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:37.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:37.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:37.639 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:37.661 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:38.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:38.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:39.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:39.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:40.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:40.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:40.662 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:40.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:41.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:41.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:41.678 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:41.701 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:42.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:42.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:42.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:42.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:44.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:44.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:44.702 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:44.702 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:11:45.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:45.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:46.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:46.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:47.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:47.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:48.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:48.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:48.764 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:48.785 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:11:48.785 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:11:48.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:48.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:11:49.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:49.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:49.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1206,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:11:49.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:11:49.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:11:50.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:50.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:50.810 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1206,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:11:50.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:11:51.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:51.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:52.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:52.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:53.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:53.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:54.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:54.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:55.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:55.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:56.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:56.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:56.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:56.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:58.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:58.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:11:59.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:11:59.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:00.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:00.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:01.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:01.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:02.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:02.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:03.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:03.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:04.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:04.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:05.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:05.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:06.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:05.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:06.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:06.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:07.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:07.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:08.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:08.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:09.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:09.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:10.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:10.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:11.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:11.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:13.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:13.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:14.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:14.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:14.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:14.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:15.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:15.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:17.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:17.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:17.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:17.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:19.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:19.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:20.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:20.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:21.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:21.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:21.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:21.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:22.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:22.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:23.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:23.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:24.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:24.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:25.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:25.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:26.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:26.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:27.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:27.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:28.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:28.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:29.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:29.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:30.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:30.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:31.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:31.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:32.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:32.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:33.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:33.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:34.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:34.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:35.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:35.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:36.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:36.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:37.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:37.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:38.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:38.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:39.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:39.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:40.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:40.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:41.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:41.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:42.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:42.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:43.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:43.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:44.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:44.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:45.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:45.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:46.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:46.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:47.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:47.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:48.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:48.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:49.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:49.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:50.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:50.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:51.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:51.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:52.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:52.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:53.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:53.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:54.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:54.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:55.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:55.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:56.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:56.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:57.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:57.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:58.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:58.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:12:59.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:12:59.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:00.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:00.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:01.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:01.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:02.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:02.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:03.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:03.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:04.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:04.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:05.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:05.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:06.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:06.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:07.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:07.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:08.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:08.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:09.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:09.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:10.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:10.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:11.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:11.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:12.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:12.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:13.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:13.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:14.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:14.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:15.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:15.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:16.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:16.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:17.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:17.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:18.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:18.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:19.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:19.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:20.843 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:13:20.844 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:13:20.844 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:13:20.844 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:13:20.844 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:13:20.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:20.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:20.851 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:13:20.851 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:13:20.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:20.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:21.832 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:21.863 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:21.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:21.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:22.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:22.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:23.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:23.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:24.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:24.865 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:24.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:24.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:25.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:25.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:25.882 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":99,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:25.915 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1206,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":99,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:26.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:26.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:27.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:27.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:28.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:28.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:28.916 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:28.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:29.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:29.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:30.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:30.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:31.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:31.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:32.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:32.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:33.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:33.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:34.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:34.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:35.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:35.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:36.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:36.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:37.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:37.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:38.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:38.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:39.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:39.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:39.947 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1212,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:39.969 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1212,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:40.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:40.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:41.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:41.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:42.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:42.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:42.970 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:42.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:43.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:43.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:43.968 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:44.000 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:44.000 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:13:44.000 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:44.000 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:13:44.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:44.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:45.006 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1216,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:13:45.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:45.037 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:13:45.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:45.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:46.053 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1216,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:13:46.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:46.072 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:13:46.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:46.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:47.080 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1216,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:13:47.102 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:13:47.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:47.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:13:47.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:47.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:48.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:13:48.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:48.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:49.221 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:13:49.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:49.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:49.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:49.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:50.177 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:50.205 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:50.205 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:13:50.205 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:13:50.205 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:50.206 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:13:50.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:50.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:51.198 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1216,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:13:51.216 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:51.216 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:13:51.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:51.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:52.213 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1216,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:13:52.236 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:52.236 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:13:52.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:52.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:53.247 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1216,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:13:53.270 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T07:13:53.270 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:13:53.270 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T07:13:53.270 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:13:53.270 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T07:13:53.271 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:13:53.291 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T07:13:53.291 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:13:53.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:53.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:53.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:53.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:54.348 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:54.378 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:54.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:54.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:55.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:55.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:56.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:56.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:57.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:13:57.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:13:57.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:57.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:58.401 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:58.420 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1216,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:13:58.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:58.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:13:59.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:13:59.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:00.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:00.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:01.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:14:01.422 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:14:02.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:02.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:03.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:03.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:04.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:04.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:05.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:05.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:06.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:06.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:07.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:07.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:08.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:08.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:08.811 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:14:08.836 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:14:08.836 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:14:08.836 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:14:08.836 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:14:09.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:09.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:09.827 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1220,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:14:09.850 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:14:09.850 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:14:10.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:10.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:10.849 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1220,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:14:10.871 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:14:11.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:11.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:12.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:12.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:13.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:13.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:14.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:14.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:15.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:14.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:16.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:16.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:17.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:17.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:17.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:17.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:18.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:18.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:19.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:19.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:20.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:20.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:21.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:21.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:22.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:22.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:23.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:23.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:24.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:24.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:25.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:25.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:27.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:27.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:28.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:28.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:29.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:29.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:30.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:30.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:31.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:31.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:32.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:32.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:33.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:33.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:34.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:34.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:35.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:35.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:36.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:36.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:37.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:37.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:38.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:38.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:39.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:39.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:39.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:39.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:41.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:40.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:41.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:41.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:43.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:43.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:43.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:43.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:45.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:45.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:46.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:46.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:47.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:47.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:48.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:48.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:49.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:49.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:50.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:50.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:51.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:51.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:52.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:52.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:53.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:53.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:54.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:54.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:55.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:55.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:56.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:56.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:57.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:57.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:58.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:58.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:14:59.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:14:59.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:00.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:00.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:01.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:01.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:02.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:02.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:03.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:03.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:04.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:04.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:05.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:05.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:06.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:06.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:07.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:07.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:08.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:08.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:09.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:09.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:10.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:10.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:11.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:11.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:12.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:12.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:13.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:13.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:14.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:14.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:15.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:15.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:16.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:16.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:17.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:17.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:18.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:18.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:19.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:19.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:20.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:20.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:21.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:21.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:22.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:22.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:23.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:23.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:24.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:24.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:25.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:25.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:26.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:26.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:27.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:27.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:28.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:28.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:29.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:29.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:30.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:30.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:31.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:31.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:32.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:32.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:33.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:33.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:34.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:34.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:34.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:34.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:35.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:35.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:36.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:36.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:37.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:37.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:38.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:38.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:39.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:39.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:40.872 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:15:40.872 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:15:40.872 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:15:40.873 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:40.873 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:15:40.881 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:15:40.881 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:15:40.881 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:40.881 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:15:41.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:41.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:41.905 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:41.938 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:42.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:42.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:43.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:43.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:44.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:44.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:44.939 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:44.939 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:15:45.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:45.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:45.941 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:45.963 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:46.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:46.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:47.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:47.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:48.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:48.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:48.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:48.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:15:49.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:49.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:49.974 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:49.996 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1220,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:50.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:50.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:51.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:51.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:52.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:52.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:52.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:52.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:15:53.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:53.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:54.000 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:54.021 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:15:54.022 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:15:54.022 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:54.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:15:54.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:54.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:55.066 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1222,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:15:55.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:55.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:55.095 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:55.095 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:15:56.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:56.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:56.077 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1222,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:15:56.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:56.099 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:15:57.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:57.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:57.075 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1222,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:15:57.097 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:15:57.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:57.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:15:58.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:58.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:58.215 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:15:59.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:15:59.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:15:59.216 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:15:59.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:15:59.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:16:00.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:00.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:00.229 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:00.259 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:00.259 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:16:00.259 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:16:00.260 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:00.260 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:16:01.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:01.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:01.270 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1222,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:16:01.303 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:01.303 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:16:02.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:02.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:02.293 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1222,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:16:02.314 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:02.314 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:16:03.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:03.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:03.326 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1222,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:16:03.347 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:16:03.348 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:16:03.348 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:16:03.348 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:16:03.348 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:16:03.348 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:16:03.370 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:16:03.371 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:16:03.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:03.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:16:04.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:04.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:04.314 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:04.345 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:05.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:05.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:06.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:06.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:07.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:07.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:07.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:07.346 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:16:08.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:08.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:08.336 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:08.367 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:09.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:09.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:10.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:10.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:11.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:11.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:11.367 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:11.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:16:12.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:12.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:12.350 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:12.391 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1222,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:13.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:13.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:14.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:14.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:15.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:15.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:15.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:15.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:16:16.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:16.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:17.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:17.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:18.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:18.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:18.875 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:18.897 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:16:18.897 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:16:18.897 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:18.897 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:16:19.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:19.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:19.884 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1224,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:16:19.905 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:16:19.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:16:20.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:20.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:20.902 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1224,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:16:20.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:16:21.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:21.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:22.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:22.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:23.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:23.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:24.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:24.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:25.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:25.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:26.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:26.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:27.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:27.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:28.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:28.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:29.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:29.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:30.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:30.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:31.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:31.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:32.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:32.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:33.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:33.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:34.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:34.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:35.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:35.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:36.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:36.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:37.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:37.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:38.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:38.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:39.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:39.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:40.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:40.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:41.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:41.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:42.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:42.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:43.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:43.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:44.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:44.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:44.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:44.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:45.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:45.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:46.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:46.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:48.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:48.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:49.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:49.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:50.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:50.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:51.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:51.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:52.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:52.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:53.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:53.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:54.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:54.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:55.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:55.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:56.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:56.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:57.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:57.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:58.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:58.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:16:59.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:16:59.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:00.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:00.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:00.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:00.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:02.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:02.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:03.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:03.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:04.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:04.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:05.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:05.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:05.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:05.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:07.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:07.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:07.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:07.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:08.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:08.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:09.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:09.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:10.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:10.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:11.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:11.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:12.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:12.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:13.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:13.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:14.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:14.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:15.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:15.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:16.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:16.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:17.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:17.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:18.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:18.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:19.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:19.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:20.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:20.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:21.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:21.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:22.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:22.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:23.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:23.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:24.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:24.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:25.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:25.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:26.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:26.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:27.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:27.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:28.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:28.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:29.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:29.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:30.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:30.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:31.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:31.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:32.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:32.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:33.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:33.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:34.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:34.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:35.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:35.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:36.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:36.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:37.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:37.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:38.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:38.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:39.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:39.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:40.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:40.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:41.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:41.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:42.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:42.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:43.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:43.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:44.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:44.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:45.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:45.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:46.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:46.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:47.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:47.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:48.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:48.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:49.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:49.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:50.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:50.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:50.932 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:17:50.932 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:17:50.932 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:17:50.933 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:17:50.933 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:17:50.939 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:17:50.939 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:17:50.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:17:50.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:17:51.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:51.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:51.944 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:17:51.966 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:17:52.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:52.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:53.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:53.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:54.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:54.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:54.967 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:17:54.967 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:17:55.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:55.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:55.983 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:17:56.006 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1224,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:17:56.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:56.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:57.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:57.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:58.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:58.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:17:59.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:17:59.008 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:17:59.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:17:59.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:00.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:00.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:01.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:01.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:02.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:02.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:03.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:03.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:04.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:04.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:05.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:05.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:06.233 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:06.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:06.258 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:18:06.258 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:06.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:18:06.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:06.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:07.289 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1232,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:18:07.320 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:07.320 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:18:07.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:07.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:08.341 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1232,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:18:08.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:08.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:18:09.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:09.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:09.467 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1232,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:18:09.489 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:18:09.489 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:09.489 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:18:10.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:10.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:10.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:18:11.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:11.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:11.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:18:11.611 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:11.611 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:18:12.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:12.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:12.607 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:12.630 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:12.631 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:18:12.631 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:18:12.631 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:12.631 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:18:13.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:13.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:13.657 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1232,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:18:13.680 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:13.680 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:18:14.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:14.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:14.671 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1232,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:18:14.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:14.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:18:15.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:15.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:15.660 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1232,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:18:15.686 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:18:15.687 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:18:15.687 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:18:15.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:18:15.687 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:18:15.687 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:18:15.707 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:18:15.708 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:18:15.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:15.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:18:16.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:16.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:16.683 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:16.706 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:17.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:17.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:18.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:18.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:19.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:19.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:19.707 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:19.707 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:18:20.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:20.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 85 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:20.706 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:20.735 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:21.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:21.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:22.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:22.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:23.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:23.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:23.736 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:23.736 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:18:24.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:24.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:24.726 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:24.755 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1232,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:25.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:25.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:26.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:26.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:27.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:27.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:27.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:27.757 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:18:28.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:28.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:29.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:29.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:30.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:30.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:31.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:31.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:31.806 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:31.828 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:18:31.828 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:18:31.828 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:31.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:18:32.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:32.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:32.796 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1234,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:18:32.828 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:18:32.828 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:18:33.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:33.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:33.841 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1234,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:18:33.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:18:34.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:34.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:35.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:35.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:36.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:36.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:37.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:37.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:38.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:38.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:39.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:39.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:40.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:40.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:41.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:41.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:42.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:42.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:43.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:43.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:44.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:44.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:45.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:45.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:46.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:46.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:47.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:47.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:48.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:48.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:49.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:49.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:50.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:50.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:51.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:51.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:52.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:52.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:53.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:53.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:54.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:54.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:55.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:55.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:56.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:56.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:57.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:57.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:58.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:58.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:18:59.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:18:59.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:00.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:00.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:01.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:01.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:02.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:02.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:03.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:03.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:04.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:04.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:05.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:05.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:06.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:06.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:07.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:07.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:08.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:08.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:09.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:09.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:10.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:10.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:11.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:11.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:12.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:12.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:13.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:13.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:14.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:14.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:15.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:15.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:16.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:16.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:17.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:17.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:18.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:18.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:19.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:19.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:20.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:20.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:21.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:21.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:22.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:22.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:23.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:23.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:24.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:24.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:25.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:25.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:26.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:26.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:27.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:27.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:28.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:28.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:29.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:29.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:30.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:30.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:31.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:31.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:32.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:32.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:33.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:33.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:34.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:34.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:35.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:35.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:36.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:36.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:37.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:37.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:38.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:38.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:39.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:39.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:40.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:40.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:41.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:41.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:42.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:42.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:43.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:43.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:44.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:44.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:45.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:45.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:46.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:46.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:47.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:47.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:48.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:48.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:49.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:49.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:50.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:50.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:51.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:51.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:52.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:52.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:53.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:53.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:54.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:54.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:55.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:55.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:56.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:56.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:57.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:57.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:58.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:58.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:19:59.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:19:59.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:00.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:00.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:01.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:01.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:02.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:02.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:03.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:03.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:03.861 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:20:03.861 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:20:03.861 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:20:03.862 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:20:03.862 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:20:03.868 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:20:03.869 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:20:03.869 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:03.869 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:04.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:04.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:04.891 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:04.914 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:05.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:05.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:06.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:06.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:07.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:07.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:07.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:07.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:08.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:08.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:08.940 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:08.970 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:09.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:09.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:10.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:10.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:11.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:11.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:11.971 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:11.971 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:12.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:12.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:12.951 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:12.983 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:13.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:13.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:14.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:14.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:15.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:15.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:15.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:15.984 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:16.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:16.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:17.009 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:17.026 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1234,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:17.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:17.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:18.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:18.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:19.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:19.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:20.028 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:20.028 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:20.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:20.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:21.011 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:21.040 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:21.040 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:20:21.040 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:21.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:20:21.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:21.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:22.033 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1238,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:20:22.064 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:22.064 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:20:22.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:22.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:23.071 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1238,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:20:23.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:23.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:20:23.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:23.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:24.078 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1238,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:20:24.099 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:20:24.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:24.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:20:24.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:24.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:25.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:20:25.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:25.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:26.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:20:26.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:26.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:26.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:26.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:27.146 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:27.176 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:27.176 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:20:27.176 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:20:27.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:27.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:20:27.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:27.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:28.130 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1238,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:20:28.152 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:28.153 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:20:28.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:28.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:29.139 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1238,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:20:29.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:29.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:20:29.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:29.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:30.142 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1238,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:20:30.164 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:20:30.164 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:20:30.164 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:20:30.164 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:20:30.164 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:20:30.165 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:20:30.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:30.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:30.217 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:20:30.217 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:20:30.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:30.217 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:31.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:31.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:31.259 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:31.287 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:32.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:32.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:33.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:33.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:34.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:34.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:34.287 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:34.288 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:35.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:35.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:35.272 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:35.295 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:36.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:36.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:37.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:37.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:38.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:38.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:38.295 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:38.296 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:39.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:39.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:39.311 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:39.329 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1238,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:40.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:40.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:41.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:41.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:42.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:42.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:42.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:42.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:20:43.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:43.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:44.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:44.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:45.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:45.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:45.866 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1242,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:45.898 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1242,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:20:45.898 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:20:45.898 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:45.899 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:20:46.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:46.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:46.886 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1242,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:20:46.909 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:20:46.909 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:20:47.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:47.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:47.900 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1242,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:20:47.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:20:48.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:48.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:49.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:49.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:50.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:50.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:51.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:51.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:52.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:52.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:53.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:53.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:54.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:54.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:55.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:55.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:56.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:56.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:57.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:57.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:58.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:58.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:20:59.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:20:59.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:00.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:00.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:01.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:01.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:02.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:02.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:03.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:03.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:04.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:04.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:05.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:05.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:06.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:06.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:07.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:07.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:08.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:08.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:09.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:09.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:10.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:10.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:11.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:11.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:12.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:12.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:13.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:13.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:14.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:14.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:15.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:15.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:16.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:16.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:17.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:17.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:18.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:18.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:19.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:19.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:20.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:20.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:21.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:21.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:22.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:22.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:23.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:23.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:24.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:24.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:25.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:25.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:26.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:26.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:27.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:27.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:28.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:28.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:29.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:29.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:30.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:30.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:31.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:31.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:32.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:32.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:33.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:33.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:34.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:34.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:35.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:35.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:36.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:36.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:37.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:37.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:38.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:38.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:39.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:39.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:40.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:40.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:41.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:41.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:42.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:42.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:43.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:43.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:44.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:44.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:45.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:45.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:46.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:46.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:47.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:47.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:48.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:48.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:49.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:49.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:50.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:50.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:51.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:51.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:52.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:52.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:53.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:53.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:54.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:54.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:55.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:55.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:56.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:56.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:57.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:57.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:58.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:58.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:21:59.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:21:59.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:00.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:00.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:01.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:01.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:02.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:02.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:03.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:03.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:04.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:04.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:05.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:05.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:06.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:06.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:07.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:07.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:08.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:08.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:09.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:09.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:10.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:10.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:11.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:11.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:12.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:12.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:13.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:13.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:14.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:14.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:15.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:15.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:16.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:16.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:17.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:17.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:17.923 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:22:17.924 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:22:17.924 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:22:17.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:17.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:22:17.932 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:22:17.932 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:22:17.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:17.933 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:18.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:18.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:18.918 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1242,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:18.947 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1242,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:19.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:19.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:20.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:20.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:21.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:21.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:21.949 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:21.949 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:22.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:22.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:23.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:23.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:24.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:24.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:25.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:25.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:26.784 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1246,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:26.809 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1246,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:26.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:26.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:27.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:27.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:28.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:28.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:29.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:29.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:29.811 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:29.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:30.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:30.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:30.811 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:30.833 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:30.833 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:22:30.833 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:30.834 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:22:31.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:31.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:31.830 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1248,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:22:31.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:31.853 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:22:32.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:32.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:32.827 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1248,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:22:32.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:32.857 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:22:33.773 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1248,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:22:33.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:33.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:33.803 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:22:33.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:33.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:22:34.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:34.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:34.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:22:35.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:35.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:35.883 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:22:35.883 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:35.884 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:36.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:36.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:36.843 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:36.864 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:36.865 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:22:36.865 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:22:36.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:36.865 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:22:37.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:37.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:37.878 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1248,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:22:37.901 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:37.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:22:38.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:38.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:38.905 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1248,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:22:38.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:38.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:22:39.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:39.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:39.856 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1248,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:22:39.878 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:22:39.879 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:22:39.879 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:22:39.879 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:22:39.879 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:22:39.879 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:22:39.898 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:22:39.898 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:22:39.898 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:39.898 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:40.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:40.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:40.903 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:40.924 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:41.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:41.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:42.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:42.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:43.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:43.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:43.925 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:43.925 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:44.924 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:44.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:44.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:44.954 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1248,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:45.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:45.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:46.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:46.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:47.955 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:47.955 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:22:47.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:47.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:49.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:49.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:50.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:50.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:51.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:51.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:52.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:52.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:53.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:53.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:54.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:54.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:54.984 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1252,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:55.015 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1252,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:22:55.016 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:22:55.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:55.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:22:55.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:55.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:56.086 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1252,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:22:56.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:22:56.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:22:56.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:56.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:57.097 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1252,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:22:57.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:57.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:57.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:22:58.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:58.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:22:59.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:22:59.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:00.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:00.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:01.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:01.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:02.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:02.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:03.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:03.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:04.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:04.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:05.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:05.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:06.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:06.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:07.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:07.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:08.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:08.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:09.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:09.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:10.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:10.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:11.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:11.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:12.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:12.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:13.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:13.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:14.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:14.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:15.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:15.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:16.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:16.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:17.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:17.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:18.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:18.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:19.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:19.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:20.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:20.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:21.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:21.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:22.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:22.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:23.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:23.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:24.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:24.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:25.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:25.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:26.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:26.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:27.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:27.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:28.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:28.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:29.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:29.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:30.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:30.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:31.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:31.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:32.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:32.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:33.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:33.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:34.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:34.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:35.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:35.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:36.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:36.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:37.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:37.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:38.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:38.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:39.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:39.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:40.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:40.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:41.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:41.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:42.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:42.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:43.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:43.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:44.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:44.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:45.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:45.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:46.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:46.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:47.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:47.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:48.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:48.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:49.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:49.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:50.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:50.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:51.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:51.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:52.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:52.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:53.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:53.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:54.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:54.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:55.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:55.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:56.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:56.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:57.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:57.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:58.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:58.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:23:59.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:23:59.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:00.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:00.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:01.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:01.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:02.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:02.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:03.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:03.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:04.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:04.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:05.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:05.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:06.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:06.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:07.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:07.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:08.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:08.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:09.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:09.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:10.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:10.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:11.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:11.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:12.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:12.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:13.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:13.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:14.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:14.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:15.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:15.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:16.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:16.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:17.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:17.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:18.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:18.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:19.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:19.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:20.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:20.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:21.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:21.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:22.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:22.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:23.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:23.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:24.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:24.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:25.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:25.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:26.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:26.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:27.127 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:24:27.127 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:24:27.127 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:24:27.128 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:27.128 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:24:27.134 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:24:27.134 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:24:27.134 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:27.134 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:27.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:27.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:28.094 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1252,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:28.116 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1252,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:28.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:28.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:29.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:29.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:30.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:30.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:31.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:31.116 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:31.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:31.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:32.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:32.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:33.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:33.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:34.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:34.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:35.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:35.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:36.023 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:36.045 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:36.045 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:24:36.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:36.046 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:24:36.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:36.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:37.039 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1258,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:24:37.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:37.061 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:24:37.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:37.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:38.064 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1258,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:24:38.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:38.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:38.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:38.086 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:24:39.073 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1258,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:24:39.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:39.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:39.095 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:24:39.096 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:39.096 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:24:40.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:40.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:40.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:24:41.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:41.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:41.154 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:24:41.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:41.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:42.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:42.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:42.147 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:42.168 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:42.168 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:24:42.169 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:24:42.169 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:42.169 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:24:43.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:43.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:43.138 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1258,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:24:43.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:43.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:24:44.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:44.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:44.161 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1258,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:24:44.191 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:44.192 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:24:45.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:45.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:45.191 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1258,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:24:45.213 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T07:24:45.214 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:24:45.214 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T07:24:45.214 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:24:45.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T07:24:45.214 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:24:45.242 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T07:24:45.242 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:24:45.242 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:45.243 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:46.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:46.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:46.323 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:46.346 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:47.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:47.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:48.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:48.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:49.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:49.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:49.347 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:49.347 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:50.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:50.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:50.331 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:50.354 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1258,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:24:51.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:51.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:52.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:52.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:53.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:53.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:53.355 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:24:53.355 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:24:54.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:54.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:55.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:55.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:56.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:56.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:57.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:57.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:58.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:58.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:24:59.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:24:59.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:00.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:00.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:00.351 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:25:00.383 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:25:00.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:25:00.384 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:25:00.384 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:25:01.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:01.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:01.374 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1262,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:25:01.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:25:01.397 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:25:02.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:02.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:02.339 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1262,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:25:02.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:25:03.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:03.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:04.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:04.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:05.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:05.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:06.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:06.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:07.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:07.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:08.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:08.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:09.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:09.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:10.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:10.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:11.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:11.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:12.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:12.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:13.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:13.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:14.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:14.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:15.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:15.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:16.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:16.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:17.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:17.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:18.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:18.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:19.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:19.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:20.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:20.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:21.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:21.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:22.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:22.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:23.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:23.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:24.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:24.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:25.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:25.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:26.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:26.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:27.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:27.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:28.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:28.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:29.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:29.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:30.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:30.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:31.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:31.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:32.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:32.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:33.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:33.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:34.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:34.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:35.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:35.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:36.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:36.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:37.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:37.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:38.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:38.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:39.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:39.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:40.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:40.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:41.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:41.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:42.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:42.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:43.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:43.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:44.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:44.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:45.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:45.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:46.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:46.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:47.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:47.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:48.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:48.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:49.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:49.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:50.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:50.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:51.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:51.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:52.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:52.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:53.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:53.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:54.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:54.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:55.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:55.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:56.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:56.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:57.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:57.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:58.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:58.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:25:59.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:25:59.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:00.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:00.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:01.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:01.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:02.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:02.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:03.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:03.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:04.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:04.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:05.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:05.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:06.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:06.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:07.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:07.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:08.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:08.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:09.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:09.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:10.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:10.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:11.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:11.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:12.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:12.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:13.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:13.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:14.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:14.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:15.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:15.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:16.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:16.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:17.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:17.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:18.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:18.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:19.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:19.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:20.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:20.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:21.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:21.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:22.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:22.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:23.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:23.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:24.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:24.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:25.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:25.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:26.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:26.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:27.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:27.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:28.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:28.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:29.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:29.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:30.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:30.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:31.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:31.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:32.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:32.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:32.364 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:26:32.364 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:26:32.364 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:26:32.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:32.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:26:32.370 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:26:32.370 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:26:32.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:32.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:26:33.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:33.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:33.345 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:33.367 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:34.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:34.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:35.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:35.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:36.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:36.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:36.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:36.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:26:37.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:37.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:37.382 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:37.405 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:38.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:38.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:39.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:39.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:40.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:40.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:40.406 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:40.406 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:26:41.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:41.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:41.418 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:41.442 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1262,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:42.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:42.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:43.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:43.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:44.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:44.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:44.442 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:44.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:26:45.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:45.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:45.394 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1264,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:45.420 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1264,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:45.421 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:26:45.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:45.421 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:26:46.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:46.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:46.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1264,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:46.436 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:46.437 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:26:47.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:47.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:47.431 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1264,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:47.460 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:47.460 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:26:48.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:48.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:48.457 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1264,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:48.480 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:26:48.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:48.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:26:49.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:49.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:49.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:26:50.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:50.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:50.538 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:26:50.539 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:50.539 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:26:51.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:51.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:51.539 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1264,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:51.570 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1264,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:26:51.570 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:26:51.570 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:26:51.570 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:51.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:26:52.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:52.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:52.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1264,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:52.620 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:52.621 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:26:53.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:53.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:53.598 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1264,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:53.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:26:53.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:26:54.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:54.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:54.654 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1264,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:26:54.676 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T07:26:54.676 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T07:26:54.676 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T07:26:54.676 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:26:54.677 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T07:26:54.677 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:26:54.691 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T07:26:54.692 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T07:26:54.692 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T07:26:54.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:26:54.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:54.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:56.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:56.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:56.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:56.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:57.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:57.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:58.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:58.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:26:59.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:26:59.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:00.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:00.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:01.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:01.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:03.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:03.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:03.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:03.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:04.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:04.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:05.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:05.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:07.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:07.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:08.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:08.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:08.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:08.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:09.693 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T07:27:09.694 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T07:27:09.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:27:09.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:27:09.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:27:09.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:09.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:10.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:10.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:11.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:11.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:12.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:12.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:13.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:13.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:14.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:14.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:15.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:15.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:16.907 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1266,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:27:16.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:16.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:16.930 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1266,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:27:16.930 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:27:16.930 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:27:16.931 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:27:17.909 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1266,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:27:17.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:17.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:17.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:27:17.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:27:18.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:18.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:18.896 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1266,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:27:18.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:27:19.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:19.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:20.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:20.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:21.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:21.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:22.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:22.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:23.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:23.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:24.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:24.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:25.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:25.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:26.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:26.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:27.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:27.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:28.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:28.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:29.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:29.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:30.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:30.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:31.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:31.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:32.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:32.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:33.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:33.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:34.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:34.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:35.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:35.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:36.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:36.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:37.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:37.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:38.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:38.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:39.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:39.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:40.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:40.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:41.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:41.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:42.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:42.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:43.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:43.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:45.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:45.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:45.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:45.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:46.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:46.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:47.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:47.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:48.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:48.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:49.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:49.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:51.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:51.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:51.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:51.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:52.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:52.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:53.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:53.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:55.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:55.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:55.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:55.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:56.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:56.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:58.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:58.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:58.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:58.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:27:59.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:27:59.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:01.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:01.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:02.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:02.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:03.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:03.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:04.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:04.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:05.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:05.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:06.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:06.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:07.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:07.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:08.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:08.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:09.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:09.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:10.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:10.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:11.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:11.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:12.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:12.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:13.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:13.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:14.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:14.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:15.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:15.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:16.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:16.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:17.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:17.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:18.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:18.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:19.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:19.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:20.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:20.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:21.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:21.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:22.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:22.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:23.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:23.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:24.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:24.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:25.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:25.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:26.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:26.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:27.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:27.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:28.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:28.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:29.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:29.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:30.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:30.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:31.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:31.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:32.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:32.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:33.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:33.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:34.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:34.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:35.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:35.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:36.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:36.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:37.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:37.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:38.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:38.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:39.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:39.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:40.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:40.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:41.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:41.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:42.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:42.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:43.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:43.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:44.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:44.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:45.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:45.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:46.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:46.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:47.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:47.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:48.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:48.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:48.926 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:28:48.926 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:28:48.926 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:28:48.927 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:28:48.927 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:28:48.933 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:28:48.933 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T07:28:48.933 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T07:28:48.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:28:49.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:49.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:50.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:50.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:51.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:51.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:52.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:52.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:53.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:53.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:54.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:54.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:55.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:55.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:56.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:56.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:57.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:57.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:58.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:58.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:28:59.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:28:59.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:00.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:00.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:01.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:01.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:02.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:02.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:03.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:03.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:03.934 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T07:29:03.935 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T07:29:03.935 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:29:03.935 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:03.935 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:29:04.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:04.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:04.945 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:04.967 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:04.968 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:29:04.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:04.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:29:05.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:05.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:05.939 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1272,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:05.960 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:05.961 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:29:06.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:06.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:06.979 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1272,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:07.010 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:07.010 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:29:07.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:07.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:08.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1272,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:08.040 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:29:08.041 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:08.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:29:08.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:08.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:09.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:29:09.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:09.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:10.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:29:10.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:10.164 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:29:10.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:10.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:11.169 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:11.199 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:11.200 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:29:11.200 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:29:11.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:11.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:29:11.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:11.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:12.173 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1272,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:12.207 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:12.208 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:29:12.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:12.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:13.235 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1272,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:13.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:13.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:13.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:13.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:29:14.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:14.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:14.232 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1272,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:14.263 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:29:14.263 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:29:14.263 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:29:14.263 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:29:14.263 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:29:14.263 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:29:14.282 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:29:14.282 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:29:14.282 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:14.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:29:15.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:15.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:15.347 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:15.370 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:16.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:16.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:17.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:17.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:18.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:18.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:18.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:18.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:29:19.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:19.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:19.379 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:19.409 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1272,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:20.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:20.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:21.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:21.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:22.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:22.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:22.411 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:22.411 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:29:23.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:23.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:24.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:24.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:25.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:25.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:26.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:26.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:27.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:27.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:28.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:28.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:28.417 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1276,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:28.450 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1276,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:29:28.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:29:28.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:28.451 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:29:29.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:29.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:29.452 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1276,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:29.483 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:29:29.484 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:29:30.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:30.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:30.489 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1276,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:29:30.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:29:31.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:31.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:32.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:32.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:33.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:33.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:34.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:34.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:35.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:35.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:36.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:36.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:37.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:37.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:38.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:38.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:39.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:39.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:40.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:40.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:41.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:41.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:42.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:42.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:43.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:43.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:44.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:44.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:45.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:45.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:46.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:46.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:47.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:47.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:48.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:48.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:49.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:49.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:50.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:50.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:51.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:51.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:52.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:52.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:53.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:53.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:54.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:54.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:55.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:55.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:56.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:56.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:57.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:57.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:58.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:58.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:29:59.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:29:59.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:00.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:00.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:01.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:01.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:02.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:02.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:03.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:03.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:04.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:04.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:05.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:05.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:06.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:06.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:07.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:07.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:08.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:08.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:09.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:09.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:10.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:10.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:11.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:11.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:12.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:12.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:13.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:13.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:14.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:14.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:15.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:15.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:16.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:16.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:17.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:17.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:18.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:18.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:19.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:19.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:20.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:20.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:21.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:21.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:22.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:22.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:23.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:23.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:24.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:24.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:25.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:25.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:26.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:26.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:27.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:27.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:28.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:28.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:29.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:29.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:30.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:30.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:31.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:31.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:32.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:32.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:33.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:33.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:34.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:34.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:35.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:35.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:36.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:36.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:37.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:37.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:38.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:38.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:39.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:39.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:40.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:40.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:41.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:41.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:42.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:42.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:43.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:43.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:44.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:44.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:45.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:45.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:46.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:46.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:47.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:47.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:48.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:48.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:49.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:49.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:50.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:50.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:51.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:51.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:52.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:52.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:53.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:53.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:54.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:54.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:55.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:55.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:56.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:56.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:57.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:57.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:58.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:58.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:30:59.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:30:59.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:00.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:00.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:00.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:31:00.512 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:31:00.513 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:31:00.513 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:00.513 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:31:00.519 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:31:00.520 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:31:00.520 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:00.520 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:31:01.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:01.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:01.527 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1276,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:01.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1276,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:02.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:02.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:03.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:03.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:04.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:04.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:04.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:04.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:31:05.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:05.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:06.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:06.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:07.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:07.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:08.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:08.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:09.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:09.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:10.072 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:10.102 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:10.102 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:31:10.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:10.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:31:10.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:10.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:11.097 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1282,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:11.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:11.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:31:11.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:11.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:12.359 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1282,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:12.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:12.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:12.388 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:12.388 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:31:13.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:13.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:13.409 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1282,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:13.439 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:31:13.439 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:13.439 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:31:14.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:14.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:14.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:31:15.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:15.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:15.573 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:31:15.574 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:15.574 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:31:16.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:16.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:16.554 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:16.584 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:16.584 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:31:16.584 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:31:16.584 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:16.585 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:31:17.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:17.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:17.602 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1282,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:17.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:17.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:31:18.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:18.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:18.622 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1282,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:18.653 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:18.653 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:31:19.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:19.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:19.669 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1282,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:19.691 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:31:19.691 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T07:31:19.691 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:31:19.692 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:31:19.692 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:31:19.692 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:31:19.707 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:31:19.707 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T07:31:19.707 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T07:31:19.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:31:20.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:20.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:21.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:21.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:22.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:22.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:23.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:23.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:24.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:24.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:25.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:25.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:26.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:26.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:27.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:27.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:28.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:28.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:29.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:29.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:30.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:30.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:31.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:31.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:32.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:32.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:33.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:33.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:34.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:34.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:34.708 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T07:31:34.709 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T07:31:34.709 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:31:34.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:34.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:31:35.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:35.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:36.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:36.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:37.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:37.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:38.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:38.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:39.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:39.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:40.373 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1286,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:40.396 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1286,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:31:40.396 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:31:40.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:40.397 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:31:40.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:40.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:41.395 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1286,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:41.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:31:41.427 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:31:41.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:41.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:42.395 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1286,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:31:42.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:31:42.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:42.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:43.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:43.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:44.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:44.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:45.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:45.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:46.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:46.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:47.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:47.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:48.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:48.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:49.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:49.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:50.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:50.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:51.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:51.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:52.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:52.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:53.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:53.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:54.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:54.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:55.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:55.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:56.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:56.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:57.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:57.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:58.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:58.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:31:59.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:31:59.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:00.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:00.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:01.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:01.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:02.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:02.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:03.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:03.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:04.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:04.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:05.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:05.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:06.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:06.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:07.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:07.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:08.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:08.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:09.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:09.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:10.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:10.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:11.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:11.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:12.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:12.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:13.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:13.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:14.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:14.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:15.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:15.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:16.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:16.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:17.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:17.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:18.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:18.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:19.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:19.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:20.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:20.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:21.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:21.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:22.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:22.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:23.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:23.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:24.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:24.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:25.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:25.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:26.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:26.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:27.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:27.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:28.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:28.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:29.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:29.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:30.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:30.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:31.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:31.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:32.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:32.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:33.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:33.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:34.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:34.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:35.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:35.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:36.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:36.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:37.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:37.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:38.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:38.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:39.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:39.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:40.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:40.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:41.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:41.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:42.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:42.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:43.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:43.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:44.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:44.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:45.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:45.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:46.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:46.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:47.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:47.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:48.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:48.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:49.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:49.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:50.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:50.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:51.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:51.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:52.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:52.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:53.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:53.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:54.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:54.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:55.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:55.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:56.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:56.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:57.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:57.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:58.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:58.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:32:59.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:32:59.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:00.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:00.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:01.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:01.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:02.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:02.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:03.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:03.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:04.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:04.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:05.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:05.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:06.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:06.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:07.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:07.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:08.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:08.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:09.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:09.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:10.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:10.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:11.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:11.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:12.418 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:33:12.419 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:33:12.419 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:33:12.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:12.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:33:12.429 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:33:12.429 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T07:33:12.429 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T07:33:12.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:33:12.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:12.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:13.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:13.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:14.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:14.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:15.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:15.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:16.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:16.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:17.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:17.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:18.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:18.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:19.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:19.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:20.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:20.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 90 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:21.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:21.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:22.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:22.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:23.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:23.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:24.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:24.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:25.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:25.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:26.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:26.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:27.430 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T07:33:27.430 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T07:33:27.431 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:33:27.431 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:27.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:27.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:27.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:28.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:28.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:29.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:29.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:30.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:30.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:31.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:31.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:32.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:32.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:33.073 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1290,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:33.095 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1290,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:33.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:33.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:34.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:34.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:35.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:35.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:36.096 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:36.096 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:36.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:36.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:37.115 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:37.145 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:37.145 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:33:37.145 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:37.145 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:33:37.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:37.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:38.164 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1294,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:38.186 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:38.186 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:33:38.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:38.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:39.430 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1294,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:39.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:39.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:33:39.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:39.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:40.430 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1294,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:40.455 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:33:40.455 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:40.456 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:33:40.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:40.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:41.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:33:41.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:41.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:42.544 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:33:42.544 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:42.545 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:42.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:42.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:43.551 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:43.573 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:43.574 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:33:43.574 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:33:43.574 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:43.574 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:33:43.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:43.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:44.571 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1294,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:44.592 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:44.593 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:33:44.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:44.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:45.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1294,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:45.618 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:45.618 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:33:45.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:45.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:46.608 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1294,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:33:46.627 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T07:33:46.627 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:33:46.628 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T07:33:46.628 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:33:46.628 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T07:33:46.628 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:33:46.647 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T07:33:46.648 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:33:46.648 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:46.648 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:46.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:46.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:47.679 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:47.710 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:47.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:47.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:48.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:48.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:49.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:49.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:50.711 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:50.711 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:50.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:50.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:51.722 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:51.749 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1294,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:33:51.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:51.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:52.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:52.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:53.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:53.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:54.750 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:33:54.750 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:33:54.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:54.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:55.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:55.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:56.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:56.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:57.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:57.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:33:58.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:58.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:00.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:33:59.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:01.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:01.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:02.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:02.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:02.264 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:34:02.286 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:34:02.287 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:34:02.287 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:34:02.287 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:34:03.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:03.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:03.182 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1298,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:34:03.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:34:03.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:34:04.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:04.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:04.223 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1298,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:34:04.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:34:05.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:05.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:06.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:06.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:07.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:07.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:08.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:08.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:09.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:09.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:10.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:10.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:11.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:11.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:12.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:12.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:13.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:13.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:14.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:14.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:15.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:15.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:16.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:16.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:17.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:17.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:18.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:18.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:19.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:19.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:20.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:20.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:21.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:21.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:22.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:22.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:23.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:23.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:24.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:24.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:25.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:25.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:26.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:26.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:27.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:27.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:28.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:28.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:29.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:29.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:30.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:30.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:31.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:31.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:32.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:32.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:33.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:33.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:34.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:34.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:35.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:35.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:36.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:36.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:37.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:37.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:38.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:38.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:39.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:39.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:40.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:40.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:41.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:41.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:42.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:42.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:43.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:43.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:44.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:44.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:45.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:45.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:46.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:46.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:47.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:47.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:48.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:48.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:49.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:49.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:50.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:50.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:51.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:51.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:52.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:52.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:53.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:53.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:54.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:54.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:55.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:55.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:56.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:56.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:57.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:57.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:58.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:58.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:34:59.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:34:59.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:00.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:00.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:01.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:01.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:02.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:02.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:03.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:03.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:04.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:04.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:05.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:05.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:06.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:06.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:07.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:07.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:08.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:08.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:09.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:09.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:10.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:10.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:11.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:11.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:12.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:12.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:13.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:13.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:14.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:14.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:15.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:15.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:16.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:16.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:17.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:17.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:18.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:18.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:19.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:19.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:20.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:20.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:21.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:21.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:22.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:22.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:23.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:23.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:24.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:24.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:25.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:25.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:26.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:26.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:27.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:27.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:28.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:28.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:29.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:29.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:30.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:30.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:31.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:31.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:32.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:32.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:33.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:33.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:34.247 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T07:35:34.247 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T07:35:34.247 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T07:35:34.247 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:34.247 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T07:35:34.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:34.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:34.253 INFO:tasks.ceph.mon.a:Started 2019-10-01T07:35:34.253 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:35:34.253 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:34.253 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:35:35.148 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:35.170 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:35.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:35.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:36.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:36.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:37.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:37.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:38.172 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:38.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:35:38.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:38.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:39.157 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:39.185 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1298,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:39.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:39.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:40.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:40.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:41.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:41.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:42.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:42.187 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:35:42.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:42.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:43.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:43.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:44.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:44.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:45.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:45.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:46.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:46.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:47.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:47.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:48.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:48.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:49.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:49.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:49.393 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:49.420 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:49.420 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:35:49.420 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:49.420 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:35:50.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:50.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:50.369 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1306,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:50.391 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:50.391 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:35:51.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:51.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:51.387 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1306,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:51.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:51.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:35:52.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:52.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:52.426 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1306,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:52.449 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:35:52.449 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:52.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:35:53.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:53.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:53.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:35:54.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:54.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:54.534 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:35:54.535 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:54.535 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:35:55.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:55.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:55.523 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:55.545 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:55.545 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:35:55.545 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:35:55.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:55.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:35:56.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:56.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:56.543 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1306,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:56.575 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:56.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:35:57.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:57.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:57.517 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1306,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:57.539 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:57.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:35:58.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:58.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:58.524 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1306,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:35:58.558 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:35:58.558 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:35:58.558 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:35:58.558 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:35:58.558 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:35:58.558 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:35:58.580 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:35:58.581 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:35:58.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:35:58.581 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:35:59.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:35:59.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:35:59.554 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:35:59.573 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:00.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:00.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:01.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:01.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:02.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:02.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:02.574 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:36:02.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:36:03.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:03.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:03.585 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:03.615 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:04.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:04.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:05.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:05.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:06.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:06.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:06.616 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:36:06.616 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:36:07.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:07.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:07.620 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:07.643 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1306,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:08.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:08.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:09.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:09.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:10.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:10.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:10.644 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:36:10.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:36:11.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:11.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:12.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:12.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:13.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:13.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:14.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:14.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:14.771 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:14.794 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:36:14.794 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:36:14.795 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:36:14.795 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:36:15.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:15.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:15.707 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1308,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:36:15.739 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:36:15.739 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:36:16.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:16.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:16.703 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1308,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:36:16.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:36:17.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:17.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:18.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:18.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:19.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:19.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:20.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:20.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:21.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:21.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:22.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:22.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:23.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:23.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:24.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:24.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:25.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:25.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:26.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:26.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:27.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:27.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:28.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:28.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:29.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:29.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:30.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:30.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:31.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:31.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:32.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:32.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:33.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:33.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:34.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:34.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:35.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:35.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:36.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:36.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:37.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:37.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:38.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:38.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:39.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:39.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:40.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:40.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:41.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:41.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:42.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:42.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:43.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:42.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:43.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:43.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:45.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:45.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:45.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:45.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:46.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:46.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:47.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:47.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:48.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:48.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:49.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:49.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:50.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:50.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:51.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:51.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:52.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:52.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:53.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:53.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:54.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:54.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:55.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:55.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:56.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:56.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:57.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:57.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:58.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:58.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:36:59.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:36:59.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:00.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:00.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:01.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:01.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:02.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:02.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:03.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:03.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:04.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:04.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:06.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:06.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:07.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:07.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:08.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:08.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:09.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:09.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:10.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:10.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:11.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:11.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:12.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:12.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:13.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:13.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:14.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:14.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:15.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:15.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:16.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:16.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:17.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:17.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:18.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:18.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:19.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:19.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:20.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:20.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:21.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:21.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:22.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:22.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:23.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:23.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:24.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:24.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:25.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:25.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:26.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:26.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:27.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:27.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:28.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:28.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:29.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:29.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:30.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:30.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:31.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:31.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:32.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:32.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:33.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:33.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:34.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:34.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:35.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:35.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:36.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:36.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:37.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:37.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:37.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:37.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:39.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:39.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:40.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:40.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:40.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:40.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:41.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:41.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:43.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:42.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:43.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:43.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:44.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:44.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:45.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:45.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:46.726 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:37:46.726 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:37:46.726 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:37:46.727 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:37:46.727 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:37:46.733 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:37:46.733 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:37:46.733 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:37:46.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:37:46.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:46.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:47.638 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:47.670 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:47.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:47.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:48.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:48.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:49.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:49.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:50.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:37:50.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:37:50.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:50.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:51.664 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:51.696 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:51.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:51.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:52.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:52.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:53.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:53.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:54.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:37:54.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:37:54.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:54.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:55.708 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":103,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:55.740 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":103,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:55.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:55.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:56.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:56.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:57.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:57.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:58.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:37:58.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:37:58.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:58.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:37:59.731 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":107,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:59.753 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1308,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":107,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:37:59.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:37:59.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:00.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:00.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:01.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:01.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:02.754 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:02.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:38:02.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:02.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:03.743 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1312,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:03.774 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1312,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:03.775 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:38:03.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:03.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:38:03.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:03.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:04.748 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1312,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:04.770 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:04.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:38:04.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:04.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:05.765 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1312,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:05.792 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:05.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:38:05.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:05.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:06.805 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1312,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:06.835 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:38:06.836 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:06.836 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:38:06.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:06.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:07.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:38:07.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:07.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:08.860 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:38:08.861 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:08.861 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:38:09.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:09.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:09.803 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1312,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:09.825 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1312,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:09.825 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:38:09.826 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:38:09.826 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:09.826 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:38:09.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:09.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:10.771 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1312,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:10.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:10.801 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:38:11.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:11.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:11.780 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1312,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:11.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:11.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:38:12.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:12.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:12.799 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1312,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:12.821 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:38:12.821 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T07:38:12.821 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:38:12.821 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:38:12.822 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:38:12.822 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:38:12.841 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:38:12.841 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T07:38:12.841 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T07:38:12.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:38:13.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:13.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:14.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:14.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:15.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:15.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:16.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:16.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:17.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:17.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:18.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:18.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:19.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:19.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:19.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:19.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:21.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:21.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:21.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:21.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:22.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:22.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:23.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:23.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:25.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:25.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:26.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:26.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:27.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:27.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:27.842 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T07:38:27.843 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T07:38:27.843 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:38:27.843 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:27.844 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:38:28.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:28.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:29.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:29.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:30.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:30.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:31.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:31.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:31.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:31.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:32.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:32.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:33.502 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1314,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:33.524 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1314,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:38:33.524 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:38:33.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:33.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:38:33.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:33.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:34.507 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1314,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:34.530 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:38:34.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:38:34.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:34.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:35.521 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1314,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:38:35.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:38:35.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:35.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:36.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:36.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:37.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:37.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:38.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:38.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:39.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:39.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:40.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:40.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:41.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:41.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:42.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:42.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:43.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:43.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:44.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:44.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:45.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:45.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:46.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:46.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:47.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:47.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:48.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:48.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:49.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:49.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:50.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:50.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:51.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:51.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:53.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:53.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:53.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:53.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:54.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:54.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:55.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:55.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:56.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:56.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:57.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:57.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:58.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:58.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:38:59.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:38:59.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:01.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:01.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:01.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:01.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:03.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:03.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:03.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:03.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:04.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:04.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:06.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:06.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:07.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:07.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:08.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:08.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:09.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:09.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:10.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:10.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:11.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:11.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:12.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:12.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:13.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:13.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:14.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:14.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:15.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:15.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:16.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:16.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:17.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:17.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:18.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:18.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:19.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:19.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:20.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:20.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:20.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:20.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:21.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:21.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:22.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:22.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:23.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:23.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:24.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:24.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:25.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:25.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:26.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:26.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:27.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:27.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:29.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:29.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:30.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:30.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:31.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:31.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:32.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:32.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:33.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:33.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:34.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:34.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:35.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:35.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:36.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:36.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:37.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:37.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:38.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:38.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:39.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:39.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:40.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:40.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:41.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:41.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:42.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:42.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:43.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:43.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:44.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:44.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:45.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:45.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:46.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:46.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:47.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:47.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:48.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:48.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:49.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:49.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:50.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:50.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:51.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:51.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:52.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:52.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:53.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:53.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:54.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:54.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:55.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:55.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:56.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:56.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:57.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:57.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:58.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:58.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:39:59.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:39:59.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:00.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:00.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:01.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:01.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:02.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:02.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:03.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:03.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:04.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:04.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:05.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:05.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:05.545 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:40:05.545 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:40:05.546 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:40:05.546 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:40:05.546 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:40:05.554 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:40:05.554 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T07:40:05.555 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T07:40:05.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T07:40:06.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:06.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:07.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:07.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:08.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:08.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:09.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:09.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:10.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:10.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:11.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:11.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:12.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:12.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:13.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:13.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:14.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:14.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:15.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:15.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:16.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:16.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:17.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:17.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:18.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:18.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:19.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:19.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:20.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:20.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:20.556 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T07:40:20.556 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T07:40:20.556 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:40:20.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:20.557 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:21.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:21.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:21.626 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:21.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:21.650 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:40:21.650 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:21.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:40:22.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:22.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:22.644 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1322,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:22.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:22.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:40:23.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:23.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:23.660 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1322,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:23.683 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:23.683 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:40:24.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:24.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:24.685 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1322,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:24.710 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:40:24.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:24.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:40:25.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:25.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:25.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:40:26.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:26.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:26.816 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:40:26.816 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:26.816 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:27.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:27.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:27.827 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:27.857 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:27.858 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:40:27.858 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:40:27.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:27.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:40:28.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:28.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:28.825 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1322,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:28.847 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:28.847 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:40:29.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:29.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:29.909 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1322,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:29.927 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:29.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:40:30.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:30.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:30.939 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1322,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:30.969 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:40:30.970 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:40:30.970 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:40:30.970 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:40:30.970 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:40:30.970 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:40:30.989 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:40:30.989 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:40:30.989 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:30.989 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:31.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:31.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:32.001 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:32.032 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:32.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:32.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:33.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:33.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:34.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:34.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:35.033 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:35.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:35.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:35.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:36.009 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:36.041 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:36.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:36.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:37.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:37.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:38.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:38.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:39.042 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:39.043 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:39.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:39.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:40.021 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:40.041 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1322,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:40.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:40.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:41.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:41.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:42.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:42.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:43.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:43.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:43.043 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:43.043 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:40:44.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:44.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:45.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:45.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:46.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:46.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:47.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:46.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:47.096 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1324,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:47.126 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1324,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:40:47.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:40:47.127 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:47.127 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:40:47.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:47.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:48.155 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1324,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:48.178 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:40:48.178 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:40:48.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:48.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:49.183 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1324,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:40:49.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:40:49.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:49.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:50.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:50.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:51.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:51.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:52.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:52.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:53.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:53.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:54.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:54.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:55.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:55.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:56.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:56.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:57.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:57.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:58.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:58.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:40:59.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:40:59.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:00.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:00.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:01.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:01.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:02.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:02.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:03.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:03.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:04.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:04.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:05.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:05.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:06.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:06.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:07.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:07.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:08.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:08.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:09.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:09.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:10.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:10.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:11.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:11.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:12.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:12.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:13.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:13.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:14.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:14.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:15.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:15.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:16.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:16.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:17.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:17.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:18.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:18.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:19.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:19.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:20.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:20.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:21.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:21.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:22.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:22.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:23.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:23.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:24.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:24.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:25.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:25.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:26.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:26.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:27.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:27.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:28.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:28.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:29.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:29.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:30.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:30.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:31.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:31.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:32.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:32.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:33.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:33.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:34.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:34.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:35.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:35.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:36.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:36.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:37.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:37.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:38.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:38.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:39.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:39.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:40.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:40.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:41.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:41.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:42.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:42.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:43.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:43.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:44.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:44.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:45.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:45.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:46.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:46.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:47.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:47.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:48.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:48.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:49.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:49.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:50.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:50.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:51.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:51.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:52.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:52.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:53.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:53.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:54.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:54.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:55.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:55.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:56.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:56.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:57.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:57.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:58.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:58.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:41:59.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:41:59.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:00.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:00.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:01.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:01.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:02.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:02.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:03.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:03.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:04.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:04.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:05.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:05.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:06.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:06.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:07.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:07.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:08.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:08.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:09.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:09.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:10.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:10.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:11.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:11.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:12.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:12.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:13.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:13.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:14.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:14.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:15.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:15.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:16.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:16.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:17.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:17.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:18.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:18.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:19.213 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:42:19.213 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:42:19.213 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:42:19.214 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:42:19.214 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:42:19.219 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:42:19.220 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:42:19.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:19.220 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:19.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:19.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:20.231 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1324,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:20.256 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1324,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:20.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:20.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:21.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:21.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:22.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:22.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:23.257 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:23.258 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:23.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:23.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:24.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:24.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:25.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:25.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:26.405 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1328,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:26.427 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1328,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:26.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:26.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:27.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:27.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:28.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:28.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:29.428 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:29.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:29.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:29.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:30.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:30.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:31.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:31.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:32.409 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1332,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:32.431 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1332,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:32.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:32.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:33.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:33.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:34.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:34.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:35.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:35.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:35.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:35.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:36.461 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1332,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:36.491 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1332,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:36.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:36.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:37.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:37.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:38.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:38.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:39.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:39.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:39.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:39.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:40.494 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:40.515 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:40.515 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:42:40.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:40.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:42:40.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:40.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:41.527 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1336,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:42:41.548 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:41.548 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:42:41.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:41.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:42.543 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1336,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:42:42.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:42.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:42:42.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:42.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:43.508 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1336,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:42:43.531 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:42:43.531 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:43.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:42:43.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:43.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:44.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:42:44.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:44.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:45.745 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:42:45.746 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:45.746 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:45.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:45.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:46.752 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:46.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:46.783 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:42:46.784 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:42:46.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:46.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:42:46.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:46.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:47.765 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1336,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:42:47.795 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:47.795 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:42:47.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:47.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:48.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1336,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:42:48.819 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:48.820 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:42:48.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:48.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:49.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1336,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:42:49.844 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:42:49.844 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:42:49.844 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:42:49.844 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:42:49.845 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:42:49.845 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:42:49.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:49.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:49.892 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:42:49.892 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:42:49.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:49.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:50.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:50.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:50.949 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:50.972 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:51.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:51.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:52.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:52.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:53.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:53.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:53.973 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:53.973 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:54.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:54.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:54.961 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:54.983 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:55.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:55.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:56.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:56.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:57.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:57.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:57.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:42:57.984 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:42:58.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:58.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:42:58.971 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:59.001 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1336,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:42:59.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:42:59.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:00.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:00.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:01.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:01.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:02.002 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:43:02.003 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:43:02.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:02.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:03.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:03.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:04.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:04.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:05.369 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1340,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:43:05.391 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1340,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:43:05.392 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:43:05.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:43:05.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:43:05.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:05.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:06.375 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1340,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:43:06.405 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:43:06.405 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:43:06.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:06.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:07.409 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1340,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:43:07.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:43:07.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:07.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:08.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:08.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:09.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:09.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:10.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:10.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:11.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:11.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:12.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:12.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:13.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:13.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:14.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:14.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:15.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:15.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:16.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:16.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:17.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:17.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:18.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:18.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:19.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:19.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:20.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:20.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:21.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:21.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:22.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:22.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:23.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:23.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:24.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:24.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:25.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:25.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:26.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:26.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:27.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:27.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:28.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:28.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:29.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:29.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:30.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:30.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:31.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:31.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:32.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:32.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:33.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:33.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:34.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:34.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:35.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:35.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:36.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:36.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:37.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:37.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:38.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:38.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:39.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:39.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:40.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:40.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:41.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:41.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:42.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:42.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:43.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:43.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:44.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:44.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:45.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:45.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:46.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:46.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:47.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:47.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:48.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:48.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:49.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:49.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:50.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:50.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:51.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:51.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:52.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:52.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:53.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:53.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:54.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:54.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:55.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:55.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:56.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:56.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:57.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:57.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:58.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:58.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:43:59.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:43:59.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:00.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:00.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:01.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:01.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:02.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:02.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:03.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:03.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:04.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:04.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:05.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:05.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:06.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:06.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:07.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:07.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:08.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:08.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:09.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:09.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:10.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:10.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:11.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:11.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:12.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:12.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:13.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:13.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:14.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:14.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:15.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:15.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:16.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:16.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:17.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:17.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:18.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:18.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:19.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:19.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:20.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:20.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:21.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:21.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:22.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:22.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:23.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:23.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:24.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:24.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:25.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:25.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:26.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:26.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:27.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:27.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:28.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:28.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:29.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:29.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:30.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:30.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:31.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:31.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:32.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:32.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:33.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:33.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:34.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:34.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:35.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:35.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:36.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:36.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:37.432 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:44:37.432 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:44:37.433 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:44:37.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:37.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:44:37.439 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:44:37.439 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:44:37.439 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:37.439 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:44:37.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:37.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:38.409 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1340,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:38.432 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1340,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:38.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:38.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:39.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:39.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:40.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:40.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:41.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:41.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:44:41.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:41.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:42.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:42.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:44.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:44.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:45.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:45.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:46.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:46.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:47.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:47.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:48.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:48.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:48.653 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:48.671 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:48.672 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:44:48.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:48.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:44:49.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:49.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:49.634 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1346,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:44:49.666 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:49.666 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:44:50.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:50.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:50.650 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1346,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:44:50.682 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:50.683 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:44:51.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:51.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:51.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1346,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:44:51.710 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:44:51.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:51.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:44:52.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:52.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:52.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:44:53.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:53.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:53.807 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:44:53.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:53.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:44:54.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:54.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:54.970 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:54.995 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:54.995 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:44:54.996 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:44:54.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:54.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:44:55.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:55.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:55.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:55.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:55.993 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1346,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:44:56.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:56.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:44:56.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:56.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:57.080 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1346,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:44:57.111 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:57.111 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:44:57.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:57.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:58.096 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1346,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:44:58.118 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:44:58.118 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:44:58.118 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:44:58.119 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:44:58.119 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:44:58.119 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:44:58.133 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:44:58.133 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:44:58.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:44:58.134 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:44:58.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:58.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:44:59.205 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:59.228 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:44:59.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:44:59.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:00.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:00.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:01.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:01.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:02.229 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:45:02.230 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:45:03.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:03.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:03.225 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:03.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:04.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:04.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:05.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:05.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:06.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:06.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:06.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:45:06.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:45:07.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:07.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:07.231 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:07.253 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1346,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:08.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:08.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:09.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:09.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:10.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:10.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:10.254 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:45:10.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:45:11.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:11.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:12.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:12.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:13.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:13.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:13.663 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1350,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:13.684 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1350,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:45:13.684 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:45:13.685 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:45:13.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:45:14.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:14.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:14.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1350,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:45:14.711 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:45:14.712 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:45:15.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:15.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:15.705 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1350,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:45:15.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:45:16.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:16.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:17.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:17.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:18.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:18.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:19.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:19.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:20.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:20.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:21.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:21.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:22.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:22.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:23.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:23.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:24.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:24.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:25.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:24.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:25.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:25.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:26.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:26.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:27.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:27.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:28.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:28.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:29.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:29.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:30.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:30.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:31.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:31.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:32.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:32.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:33.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:33.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:34.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:34.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:35.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:35.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:36.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:36.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:37.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:37.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:38.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:38.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:39.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:39.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:40.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:40.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:41.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:41.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:43.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:43.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:44.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:44.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:44.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:44.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:45.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:45.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:47.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:47.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:48.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:48.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:49.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:49.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:50.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:50.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:51.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:51.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:51.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:51.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:52.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:52.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:53.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:53.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:54.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:54.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:55.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:55.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:56.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:56.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:57.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:57.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:58.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:58.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:45:59.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:45:59.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:00.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:00.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:02.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:02.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:02.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:02.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:03.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:03.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:04.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:04.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:05.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:05.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:06.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:06.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:07.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:07.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:08.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:08.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:09.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:09.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:10.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:10.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:11.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:11.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:12.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:12.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:13.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:13.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:14.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:14.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:16.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:16.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:17.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:17.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:18.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:18.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:19.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:19.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:20.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:20.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:21.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:21.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:22.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:22.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:23.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:23.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:24.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:24.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:25.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:25.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:26.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:26.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:27.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:27.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:28.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:27.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:29.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:29.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:30.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:30.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:30.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:30.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:31.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:31.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:32.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:32.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:34.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:34.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:34.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:34.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:35.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:35.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:36.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:36.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:37.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:37.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:39.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:38.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:39.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:39.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:40.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:40.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:42.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:41.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:43.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:43.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:44.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:44.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:45.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:45.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:45.728 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:46:45.729 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:46:45.729 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:46:45.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:45.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:46:45.737 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:46:45.737 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:46:45.737 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:45.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:46:46.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:46.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:46.735 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1350,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:46:46.765 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1350,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:46:47.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:47.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:48.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:47.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:48.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:48.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:49.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:49.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:46:49.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:49.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:51.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:51.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:52.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:52.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:53.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:53.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:54.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:54.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:54.665 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:46:54.697 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:46:54.698 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:46:54.698 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:54.698 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:46:55.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:55.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:55.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1356,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:46:55.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:55.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:46:56.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:56.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:56.993 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1356,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:46:57.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:57.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:46:57.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:57.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:58.029 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1356,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:46:58.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:58.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:58.059 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:46:58.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:46:58.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:46:59.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:46:59.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:46:59.172 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:47:00.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:00.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:00.173 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:47:00.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:00.174 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:47:01.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:01.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:01.190 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:01.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:01.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:47:01.220 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:47:01.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:01.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:47:02.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:02.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:02.236 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1356,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:47:02.268 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:02.268 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:47:03.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:03.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:03.264 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1356,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:47:03.286 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:03.286 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:47:04.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:04.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:04.288 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1356,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:47:04.310 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:47:04.310 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:47:04.310 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:47:04.310 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:47:04.310 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:47:04.311 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:47:04.329 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:47:04.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:47:04.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:04.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:47:05.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:05.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:05.377 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:05.399 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:06.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:06.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:07.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:07.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:08.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:08.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:08.400 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:08.400 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:47:09.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:09.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:09.408 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:09.430 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:10.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:10.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:11.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:11.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:12.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:12.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:12.431 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:12.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:47:13.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:13.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:13.410 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:13.432 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1356,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:14.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:14.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:15.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:15.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:16.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:16.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:16.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:16.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:47:17.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:17.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:18.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:18.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:19.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:19.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:19.315 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1360,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:19.343 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1360,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:47:19.343 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:47:19.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:19.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:47:20.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:20.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:20.371 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1360,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:47:20.398 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:47:20.399 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:47:21.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:21.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:21.395 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1360,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:47:21.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:47:22.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:22.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:23.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:23.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:24.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:24.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:25.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:25.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:26.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:26.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:27.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:27.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:28.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:28.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:29.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:29.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:30.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:30.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:31.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:31.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:32.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:32.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:33.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:33.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:34.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:34.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:35.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:35.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:36.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:36.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:37.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:37.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:38.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:38.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:39.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:39.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:40.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:40.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:41.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:41.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:42.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:42.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:43.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:43.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:44.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:44.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:45.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:45.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:46.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:46.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:47.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:47.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:48.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:48.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:49.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:49.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:50.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:50.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:51.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:51.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:52.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:52.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:53.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:53.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:54.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:54.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:55.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:55.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:56.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:56.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:57.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:57.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:58.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:58.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:47:59.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:47:59.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:00.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:00.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:01.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:01.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:02.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:02.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:03.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:03.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:04.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:04.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:05.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:05.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:06.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:06.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:07.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:07.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:08.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:08.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:09.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:09.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:10.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:10.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:11.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:11.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:12.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:12.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:13.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:13.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:14.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:14.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:15.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:15.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:16.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:16.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:17.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:17.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:18.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:18.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:19.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:19.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:20.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:20.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:21.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:21.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:22.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:22.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:23.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:23.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:24.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:24.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:25.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:25.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:26.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:26.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:27.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:27.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:28.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:28.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:29.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:29.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:30.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:30.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:31.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:31.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:32.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:32.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:33.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:33.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:34.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:34.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:35.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:35.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:36.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:36.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:37.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:37.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:38.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:38.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:39.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:39.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:40.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:40.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:41.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:41.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:42.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:42.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:43.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:43.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:44.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:44.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:45.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:45.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:46.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:46.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:47.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:47.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:48.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:48.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:49.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:49.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:50.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:50.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:51.421 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:48:51.422 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:48:51.422 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:48:51.422 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:48:51.422 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:48:51.428 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:48:51.428 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:48:51.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:48:51.429 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:48:51.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:51.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:52.392 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1360,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:48:52.428 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1360,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:48:52.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:52.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:53.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:53.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:54.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:54.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:55.429 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:48:55.429 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:48:55.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:55.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:56.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:56.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:57.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:57.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:58.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:58.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:48:59.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:48:59.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:00.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:00.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:01.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:01.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:02.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:02.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:02.685 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:02.707 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:02.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:49:02.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:02.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:49:03.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:03.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:03.708 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1366,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:49:03.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:03.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:49:04.351 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T07:49:04.341+0000 7f5d7ed9d700 0 --2- 172.21.6.124:0/2879237522 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f5d8007fcd0 0x7f5d800800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T07:49:04.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:04.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:04.987 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1366,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:49:05.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:05.008 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:49:05.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:05.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:06.032 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1366,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:49:06.053 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:49:06.054 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:06.054 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:49:06.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:06.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:07.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:49:07.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:07.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:08.184 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:49:08.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:08.184 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:49:08.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:08.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:09.258 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:09.283 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:09.283 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:49:09.283 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:49:09.284 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:09.284 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:49:09.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:09.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:10.289 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1366,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:49:10.318 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:10.318 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:49:10.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:10.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:11.345 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1366,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:49:11.367 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:11.367 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:49:11.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:11.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:12.331 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1366,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:49:12.353 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:49:12.353 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:49:12.354 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:49:12.354 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:49:12.354 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:49:12.354 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:49:12.371 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:49:12.371 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:49:12.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:12.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:49:12.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:12.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:13.424 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:13.449 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:13.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:13.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:14.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:14.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:15.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:15.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:16.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:16.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:49:16.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:16.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:17.424 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:17.446 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:17.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:17.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:18.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:18.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:19.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:19.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:20.447 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:20.448 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:49:20.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:20.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:21.462 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:21.493 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1366,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:21.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:21.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:22.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:22.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:23.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:23.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:24.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:24.494 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:49:24.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:24.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:25.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:25.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:26.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:26.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:27.548 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:27.571 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:49:27.571 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:49:27.572 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:27.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:49:27.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:27.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:28.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:28.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:28.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1370,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:49:28.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:49:28.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:49:29.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:29.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:29.607 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1370,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:49:29.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:49:30.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:30.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:31.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:31.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:32.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:32.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:33.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:33.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:34.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:34.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:35.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:35.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:36.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:36.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:37.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:37.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:38.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:38.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:39.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:39.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:40.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:40.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:41.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:41.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:42.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:42.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:43.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:43.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:44.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:44.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:45.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:45.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:46.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:46.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:47.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:47.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:48.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:48.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:49.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:49.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:50.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:50.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:51.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:51.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:52.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:52.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:53.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:53.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:54.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:54.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:55.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:55.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:56.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:56.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:57.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:57.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:58.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:58.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:49:59.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:49:59.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:00.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:00.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:01.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:01.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:02.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:02.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:03.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:03.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:04.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:04.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:05.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:05.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:06.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:06.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:07.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:07.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:08.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:08.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:09.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:09.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:10.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:10.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:11.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:11.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:12.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:12.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:13.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:13.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:14.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:14.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:15.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:15.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:16.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:16.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:17.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:17.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:18.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:18.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:19.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:19.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:20.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:20.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:21.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:21.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:22.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:22.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:23.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:23.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:24.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:24.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:25.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:25.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:26.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:26.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:27.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:27.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:28.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:28.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:29.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:29.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:30.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:30.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:31.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:31.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:32.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:32.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:33.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:33.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:34.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:34.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:35.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:35.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:36.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:36.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:37.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:37.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:38.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:38.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:39.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:39.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:40.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:40.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:41.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:41.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:42.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:42.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:43.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:43.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:44.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:44.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:45.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:45.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:46.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:46.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:47.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:47.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:48.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:48.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:49.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:49.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:50.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:50.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:51.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:51.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:52.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:52.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:53.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:53.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:54.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:54.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:55.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:55.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:56.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:56.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:57.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:57.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:58.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:58.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:59.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:50:59.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:50:59.627 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:50:59.627 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:50:59.627 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:50:59.628 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:50:59.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:50:59.635 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:50:59.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:50:59.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:50:59.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:00.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:00.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:00.653 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:00.683 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1370,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:01.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:01.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:02.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:02.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:03.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:03.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:03.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:03.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:04.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:04.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:05.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:05.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:06.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:06.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:07.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:07.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:08.474 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:08.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:08.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:08.495 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:08.496 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:51:08.496 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:08.496 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:51:09.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:09.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:09.507 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1376,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:51:09.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:09.537 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:51:10.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:10.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:10.535 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T07:51:10.523+0000 7f7832d13700 0 --2- 172.21.6.124:0/820737554 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f782c07fcd0 0x7f782c0800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T07:51:11.169 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1376,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:51:11.190 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:11.191 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:51:11.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:11.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:12.197 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1376,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:51:12.220 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:51:12.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:12.220 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:51:12.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:12.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:13.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:51:13.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:13.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:14.314 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:51:14.314 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:14.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:14.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:14.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:15.322 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:15.342 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:15.342 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:51:15.342 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:51:15.342 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:15.342 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:51:15.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:15.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:16.257 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1376,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:51:16.287 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:16.287 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:51:16.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:16.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:17.300 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1376,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:51:17.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:17.331 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:51:17.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:17.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:18.338 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1376,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:51:18.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:18.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:18.368 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:51:18.369 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:51:18.369 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:51:18.369 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:51:18.369 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:51:18.369 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:51:18.388 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:51:18.388 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:51:18.388 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:18.389 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:19.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:19.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:19.461 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:19.492 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:20.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:20.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:21.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:21.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:22.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:22.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:22.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:22.494 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:23.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:23.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:23.519 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:23.549 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:24.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:24.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:25.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:25.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:26.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:26.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:26.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:26.551 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:27.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:27.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:27.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:27.546 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1376,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:28.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:28.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:29.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:29.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:30.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:30.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:30.547 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:30.548 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:51:31.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:31.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:32.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:32.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:33.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:33.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:33.400 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1380,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:33.432 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1380,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:51:33.432 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:51:33.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:33.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:51:34.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:34.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:34.449 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1380,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:51:34.472 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:51:34.472 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:51:35.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:35.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:35.468 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1380,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:51:35.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:51:36.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:36.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:37.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:37.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:38.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:38.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:39.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:39.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:40.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:40.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:41.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:41.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:42.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:42.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:43.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:43.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:44.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:44.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:45.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:45.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:46.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:46.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:47.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:47.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:48.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:48.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:49.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:49.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:50.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:50.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:51.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:51.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:52.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:52.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:53.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:53.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:54.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:54.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:55.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:55.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:56.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:56.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:57.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:57.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:58.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:58.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:51:59.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:51:59.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:00.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:00.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:01.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:01.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:02.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:02.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:03.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:03.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:04.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:04.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:05.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:05.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:06.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:06.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:07.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:07.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:08.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:08.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:09.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:09.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:10.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:10.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:11.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:11.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:12.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:12.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:13.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:13.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:14.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:14.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:15.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:15.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:16.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:16.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:17.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:17.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:18.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:18.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:19.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:19.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:20.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:20.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:21.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:21.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:22.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:22.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:23.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:23.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:24.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:24.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:25.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:25.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:26.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:26.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:27.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:27.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:28.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:28.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:29.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:29.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:30.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:30.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:31.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:31.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:32.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:32.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:33.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:33.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:34.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:34.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:35.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:35.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:36.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:36.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:37.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:37.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:38.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:38.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:39.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:39.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:40.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:40.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:41.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:41.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:42.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:42.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:43.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:43.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:44.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:44.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:45.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:45.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:46.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:46.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:47.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:47.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:48.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:48.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:49.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:49.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:50.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:50.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:51.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:51.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:52.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:52.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:53.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:53.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:54.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:54.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:55.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:55.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:56.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:56.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:57.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:57.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:58.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:58.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:52:59.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:52:59.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:00.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:00.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:01.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:01.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:02.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:02.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:03.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:03.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:04.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:04.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:05.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:05.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:05.499 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:53:05.500 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:53:05.500 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:53:05.500 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:05.500 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:53:05.506 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:53:05.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:53:05.507 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:05.507 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:06.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:06.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:06.495 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1380,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:06.517 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1380,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:07.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:07.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:08.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:08.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:09.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:09.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:09.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:09.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:10.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:10.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:11.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:11.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:12.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:12.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:13.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:13.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:14.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:14.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:14.416 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:14.438 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:14.438 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:53:14.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:14.439 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:53:15.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:15.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:15.540 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1386,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:53:15.563 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:15.563 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:53:16.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:16.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:16.582 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1386,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:53:16.615 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:16.615 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:53:17.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:17.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:17.607 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1386,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:53:17.629 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:53:17.629 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:17.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:53:18.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:18.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:18.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:53:19.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:19.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:19.723 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:53:19.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:19.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:20.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:20.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:20.710 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:20.741 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:20.741 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:53:20.741 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:53:20.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:20.742 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:53:21.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:21.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:21.749 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1386,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:53:21.780 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:21.780 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:53:22.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:22.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:22.786 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1386,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:53:22.809 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:22.809 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:53:23.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:23.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:23.883 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1386,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:53:23.905 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:53:23.905 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:53:23.905 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:53:23.905 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:53:23.905 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:53:23.906 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:53:23.924 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:53:23.924 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:53:23.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:23.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:24.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:24.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:24.979 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:25.001 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:25.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:25.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:26.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:26.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:27.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:27.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:28.006 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:28.006 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:28.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:28.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:29.010 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:29.033 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1386,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:29.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:29.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:30.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:30.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:31.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:31.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:32.034 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:32.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:53:32.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:32.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:33.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:33.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:34.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:34.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:35.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:35.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:36.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:36.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:37.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:37.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:38.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:38.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:38.645 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1390,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:38.674 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1390,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:53:38.674 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:53:38.674 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:38.674 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:53:39.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:39.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:39.674 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1390,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:53:39.696 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:53:39.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:53:40.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:40.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:40.691 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1390,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:53:40.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:53:41.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:41.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:42.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:42.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:43.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:43.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:44.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:44.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:45.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:45.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:46.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:46.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:47.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:47.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:48.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:48.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:49.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:49.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:50.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:50.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:51.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:51.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:52.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:52.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:53.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:53.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:54.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:54.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:55.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:55.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:56.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:56.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:57.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:57.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:58.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:58.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:53:59.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:53:59.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:00.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:00.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:01.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:01.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:02.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:02.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:03.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:03.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:04.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:04.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:05.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:05.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:06.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:06.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:07.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:07.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:08.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:08.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:09.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:09.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:10.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:10.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:11.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:11.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:12.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:12.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:13.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:13.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:14.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:14.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:15.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:15.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:16.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:16.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:17.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:17.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:18.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:18.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:19.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:19.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:20.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:20.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:21.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:21.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:22.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:22.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:23.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:23.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:24.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:24.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:25.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:25.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:26.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:26.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:27.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:27.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:28.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:28.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:29.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:29.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:30.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:30.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:31.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:31.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:32.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:32.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:33.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:33.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:34.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:34.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:35.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:35.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:36.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:36.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:37.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:37.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:38.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:38.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:39.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:39.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:40.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:40.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:41.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:41.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:42.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:42.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:43.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:43.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:44.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:44.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:45.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:45.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:46.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:46.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:47.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:47.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:48.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:48.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:49.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:49.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:50.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:50.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:51.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:51.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:52.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:52.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:53.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:53.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:54.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:54.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:55.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:55.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:56.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:56.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:57.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:57.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:58.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:58.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:54:59.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:54:59.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:00.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:00.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:01.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:01.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:02.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:02.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:03.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:03.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:04.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:04.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:05.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:05.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:06.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:06.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:07.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:07.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:08.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:08.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:09.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:09.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:10.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:10.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:10.714 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:55:10.714 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:55:10.715 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:55:10.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:10.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:55:10.721 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:55:10.721 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:55:10.721 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:10.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:11.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:11.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:11.709 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1390,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:11.731 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1390,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:12.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:12.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:13.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:13.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:14.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:14.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:14.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:14.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:15.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:15.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:16.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:16.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:17.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:17.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:18.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:18.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:19.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:19.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:19.648 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:19.678 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:19.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:55:19.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:19.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:55:20.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:20.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:20.654 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1396,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:55:20.686 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:20.686 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:55:21.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:21.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:21.917 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1396,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:55:21.947 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:21.948 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:55:22.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:22.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:22.965 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1396,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:55:22.988 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:55:22.988 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:22.988 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:55:23.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:23.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:24.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:55:24.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:24.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:25.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:55:25.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:25.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:25.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:25.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:26.106 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:26.137 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:26.137 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:55:26.137 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:55:26.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:26.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:55:26.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:26.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:27.150 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1396,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:55:27.172 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:27.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:55:27.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:27.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:28.182 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1396,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:55:28.212 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:28.212 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:55:28.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:28.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:29.214 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1396,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:55:29.232 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:55:29.232 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:55:29.232 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:55:29.232 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:55:29.233 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:55:29.233 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:55:29.254 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:55:29.255 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:55:29.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:29.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:29.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:29.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:30.168 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:30.190 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:30.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:30.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:31.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:31.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:32.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:32.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:33.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:33.192 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:33.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:33.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:34.195 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:34.217 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:34.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:34.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:35.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:35.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:36.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:36.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:37.218 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:37.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:37.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:37.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:38.240 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:38.269 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1396,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:38.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:38.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:39.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:39.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:40.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:40.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:41.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:41.270 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:55:41.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:41.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:42.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:42.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:43.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:43.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:43.891 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:43.922 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:55:43.922 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:55:43.923 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:43.923 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:55:44.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:44.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:44.843 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1398,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:55:44.873 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:55:44.873 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:55:45.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:45.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:45.872 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1398,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:55:45.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:55:46.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:46.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:47.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:47.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:48.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:48.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:49.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:49.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:50.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:50.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:51.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:51.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:52.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:52.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:53.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:53.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:54.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:54.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:55.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:55.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:56.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:56.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:57.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:57.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:58.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:58.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:55:59.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:55:59.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:00.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:00.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:01.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:01.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:02.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:02.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:03.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:03.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:04.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:04.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:05.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:05.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:06.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:06.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:07.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:07.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:08.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:08.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:09.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:09.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:10.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:10.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:11.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:11.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:12.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:12.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:13.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:13.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:14.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:14.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:15.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:15.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:16.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:16.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:17.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:17.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:18.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:18.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:19.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:19.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:20.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:20.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:21.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:21.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:22.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:22.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:23.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:23.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:24.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:24.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:25.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:25.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:26.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:26.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:27.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:27.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:29.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:28.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:30.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:30.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:31.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:31.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:32.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:32.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:33.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:33.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:34.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:34.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:35.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:35.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:35.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:35.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:36.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:36.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:37.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:37.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:38.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:38.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:39.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:39.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:40.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:40.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:41.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:41.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:42.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:42.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:43.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:43.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:44.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:44.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:46.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:46.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:47.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:47.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:48.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:48.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:49.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:49.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:50.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:50.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:51.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:51.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:52.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:52.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:53.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:53.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:54.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:54.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:55.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:55.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:56.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:56.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:57.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:57.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:58.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:58.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:59.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:59.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:56:59.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:56:59.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:01.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:01.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:02.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:02.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:03.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:03.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:04.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:04.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:05.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:05.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:06.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:06.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:07.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:07.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:08.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:08.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:09.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:09.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:10.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:10.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:11.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:11.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:11.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:11.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:12.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:12.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:13.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:13.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:14.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:14.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:15.891 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:57:15.891 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T07:57:15.891 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T07:57:15.892 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T07:57:15.892 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T07:57:15.898 INFO:tasks.ceph.mon.b:Started 2019-10-01T07:57:15.898 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:57:15.898 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:15.898 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:15.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:15.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:16.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:16.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:16.935 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:16.957 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:17.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:17.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:18.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:18.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:19.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:19.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:19.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:19.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:20.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:20.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:20.991 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:21.015 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:21.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:21.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:22.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:22.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:23.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:23.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:24.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:24.017 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:24.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:24.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:25.018 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:25.047 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:25.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:25.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:26.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:26.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:27.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:27.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:28.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:28.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:28.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:28.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:29.068 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:29.087 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1398,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:29.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:29.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:31.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:31.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:31.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:31.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:32.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:32.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:32.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:32.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:33.041 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:33.072 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:33.072 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:57:33.073 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:33.073 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:57:33.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:33.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:34.088 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1402,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:57:34.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:34.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:57:34.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:34.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:35.101 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1402,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:57:35.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:35.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:57:35.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:35.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:36.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1402,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:57:36.154 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:57:36.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:36.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:57:36.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:36.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:37.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:57:37.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:37.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:38.237 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:57:38.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:38.237 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:38.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:38.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:39.258 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:39.288 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:39.288 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:57:39.288 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:57:39.288 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:39.289 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:57:39.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:39.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:40.292 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1402,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:57:40.322 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:40.322 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:57:40.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:40.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:41.351 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1402,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:57:41.383 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:41.384 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:57:41.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:41.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:42.378 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1402,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:57:42.401 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T07:57:42.401 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:57:42.401 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T07:57:42.401 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:57:42.402 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T07:57:42.402 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:57:42.421 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T07:57:42.421 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:57:42.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:42.421 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:42.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:42.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:43.484 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:43.506 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:43.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:43.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:44.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:44.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:45.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:45.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:46.507 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:46.507 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:46.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:46.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:47.494 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:47.523 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:47.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:47.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:48.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:48.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:49.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:49.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:50.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:50.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:50.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:50.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:51.514 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:51.536 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1402,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:51.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:51.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:52.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:52.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:53.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:53.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:54.537 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:54.538 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:57:54.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:54.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:55.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:55.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:56.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:56.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:57.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:57.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:57.942 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1406,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:57.965 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1406,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:57:57.965 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T07:57:57.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:57.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:57:58.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:58.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:58.972 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1406,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:57:58.994 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:57:58.994 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:57:59.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:57:59.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:57:59.984 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1406,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:58:00.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T07:58:00.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:00.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:01.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:01.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:02.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:02.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:03.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:03.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:04.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:04.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:05.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:05.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:06.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:06.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:07.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:07.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:08.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:08.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:09.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:09.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:10.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:10.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:11.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:11.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:12.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:12.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:13.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:13.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:14.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:14.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:15.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:15.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:16.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:16.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:17.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:17.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:19.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:19.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:20.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:20.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:21.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:21.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:21.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:21.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:22.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:22.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:24.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:24.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:24.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:24.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:26.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:26.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:26.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:26.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:27.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:27.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:28.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:28.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:29.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:29.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:30.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:30.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:31.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:31.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:32.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:32.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:33.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:33.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:34.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:34.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:35.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:35.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:36.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:36.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:37.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:37.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:38.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:38.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:39.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:39.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:40.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:40.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:41.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:41.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:42.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:42.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:43.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:43.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:44.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:44.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:45.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:45.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:46.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:46.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:47.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:47.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:48.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:48.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:49.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:49.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:50.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:50.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:51.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:51.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:52.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:52.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:53.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:53.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:54.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:54.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:55.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:55.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:56.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:56.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:57.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:57.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:58.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:58.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:58:59.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:58:59.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:00.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:00.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:01.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:01.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:02.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:02.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:03.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:03.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:04.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:04.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:05.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:05.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:06.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:06.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:07.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:07.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:08.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:08.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:09.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:09.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:10.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:10.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:11.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:11.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:12.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:12.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:13.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:13.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:14.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:14.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:15.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:15.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:16.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:16.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:17.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:17.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:18.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:18.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:19.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:19.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:20.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:20.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:21.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:21.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:22.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:22.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:23.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:23.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:24.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:24.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:25.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:25.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:26.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:26.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:27.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:27.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:28.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:28.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:29.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:29.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:30.015 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T07:59:30.015 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T07:59:30.015 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T07:59:30.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:30.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T07:59:30.025 INFO:tasks.ceph.mon.c:Started 2019-10-01T07:59:30.025 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:59:30.025 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:30.026 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:30.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:30.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:31.026 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1406,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:31.047 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1406,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:31.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:31.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:32.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:32.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:33.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:33.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:34.048 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:34.048 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:34.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:34.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:35.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:35.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:36.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:36.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:37.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:37.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:38.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:38.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:38.856 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:38.887 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:38.887 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:59:38.887 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:38.888 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:59:39.881 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1412,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:59:39.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:39.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:39.906 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:39.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:59:40.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:40.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:41.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1412,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:59:41.562 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:41.562 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:59:41.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:41.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:42.577 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1412,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:59:42.607 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T07:59:42.607 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:42.607 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T07:59:42.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:42.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:43.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T07:59:43.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:43.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:44.721 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T07:59:44.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:44.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:44.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:44.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:45.714 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:45.736 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:45.736 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T07:59:45.736 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T07:59:45.736 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:45.737 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T07:59:46.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:46.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:46.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1412,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T07:59:46.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:46.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T07:59:47.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:47.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:47.730 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1412,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:59:47.751 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:47.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T07:59:48.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:48.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:48.736 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1412,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T07:59:48.758 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T07:59:48.759 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T07:59:48.759 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T07:59:48.759 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T07:59:48.759 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T07:59:48.759 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T07:59:48.786 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T07:59:48.786 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T07:59:48.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:48.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:49.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:49.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:49.765 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:49.797 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:50.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:50.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:51.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:51.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:52.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:52.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:52.798 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:52.798 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:53.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:53.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:53.783 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:53.801 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:54.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:54.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:55.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:55.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:56.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:56.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:56.802 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T07:59:56.802 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T07:59:57.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:57.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:57.829 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:57.855 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1412,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T07:59:58.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:58.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T07:59:59.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T07:59:59.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:00.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:00.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:00.856 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:00:00.856 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:00:01.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:01.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:02.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:02.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:03.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:03.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:04.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:04.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:04.913 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:00:04.937 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:00:04.937 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:00:04.937 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:00:04.937 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:00:05.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:05.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:05.995 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1414,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:00:06.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:05.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:06.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:00:06.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:00:06.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:06.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:07.025 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1414,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:00:07.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:00:07.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:07.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:08.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:08.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:09.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:09.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:10.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:10.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:11.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:11.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:12.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:12.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:13.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:13.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:14.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:14.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:15.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:15.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:16.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:16.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:17.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:17.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:18.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:18.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:19.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:19.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:20.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:20.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:21.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:21.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:22.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:22.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:23.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:23.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:24.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:24.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:25.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:25.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:26.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:26.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:27.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:27.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:28.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:28.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:29.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:29.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:30.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:30.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:31.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:31.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:32.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:32.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:33.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:33.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:34.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:34.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:35.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:35.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:36.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:36.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:37.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:37.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:38.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:38.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:39.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:39.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:40.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:40.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:41.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:41.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:42.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:42.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:43.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:43.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:44.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:44.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:45.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:45.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:46.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:46.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:47.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:47.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:48.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:48.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:49.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:49.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:50.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:50.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:51.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:51.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:52.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:52.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:53.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:53.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:54.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:54.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:55.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:55.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:56.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:56.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:57.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:57.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:58.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:58.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:00:59.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:00:59.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:00.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:00.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:01.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:01.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:02.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:02.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:03.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:03.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:04.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:04.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:05.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:05.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:06.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:06.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:07.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:07.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:08.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:08.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:09.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:09.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:10.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:10.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:11.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:11.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:12.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:12.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:13.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:13.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:14.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:14.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:15.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:15.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:16.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:16.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:17.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:17.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:18.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:18.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:19.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:19.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:20.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:20.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:21.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:21.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:22.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:22.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:23.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:23.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:24.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:24.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:25.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:25.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:26.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:26.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:27.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:27.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:28.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:28.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:29.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:29.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:30.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:30.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:31.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:31.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:33.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:33.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:34.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:34.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:34.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:34.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:36.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:36.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:37.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:37.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:37.058 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:01:37.058 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:01:37.058 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:01:37.059 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:01:37.059 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:01:37.072 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:01:37.072 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:01:37.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:37.072 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:01:37.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:37.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:38.086 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:38.103 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:39.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:39.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:40.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:40.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:41.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:41.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:41.105 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:41.105 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:01:42.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:42.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:42.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:42.106 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:43.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:43.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:44.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:44.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:45.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:45.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:45.107 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:45.107 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:01:46.094 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:46.124 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:46.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:46.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:47.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:47.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:48.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:48.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:49.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:49.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:49.126 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:49.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:01:50.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:50.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:50.104 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:50.132 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1414,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:51.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:51.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:52.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:52.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:53.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:53.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:53.132 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:53.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:01:54.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:54.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:54.143 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:54.170 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:01:54.170 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:01:54.170 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:54.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:01:55.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:55.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:55.194 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1418,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:01:55.224 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:55.224 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:01:56.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:56.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:56.195 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1418,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:01:56.225 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:56.225 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:01:57.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:57.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:57.208 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1418,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:01:57.230 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:01:57.231 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:57.231 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:01:58.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:58.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:58.342 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:01:59.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:01:59.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:01:59.342 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:01:59.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:01:59.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:02:00.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:00.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:00.335 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:00.367 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:00.367 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:02:00.367 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:02:00.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:00.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:02:01.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:01.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:01.327 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1418,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:02:01.357 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:01.357 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:02:02.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:02.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:02.367 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1418,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:02:02.398 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:02.398 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:02:03.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:03.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:03.378 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1418,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:02:03.400 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:02:03.401 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:02:03.401 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:02:03.401 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:02:03.402 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:02:03.402 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:02:03.416 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:02:03.416 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:02:03.417 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:03.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:02:04.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:04.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:04.467 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:04.489 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:05.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:05.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:06.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:06.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:07.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:07.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:07.490 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:07.490 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:02:08.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:08.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:08.487 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:08.509 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1418,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:09.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:09.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:10.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:10.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:11.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:11.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:11.511 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:11.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:02:12.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:12.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:13.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:13.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:14.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:14.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:15.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:15.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:16.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:16.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:17.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:17.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:17.508 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1422,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:17.541 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1422,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:02:17.541 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:02:17.541 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:17.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:02:18.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:18.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:18.540 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1422,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:02:18.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:02:18.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:02:19.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:19.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:19.549 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1422,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:02:19.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:02:20.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:20.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:21.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:21.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:22.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:22.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:23.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:23.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:24.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:24.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:25.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:25.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:26.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:26.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:27.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:27.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:28.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:28.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:29.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:29.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:30.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:30.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:31.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:31.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:32.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:32.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:33.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:33.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:34.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:34.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:35.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:35.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:36.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:36.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:37.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:37.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:38.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:38.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:39.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:39.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:40.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:40.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:41.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:41.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:42.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:42.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:43.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:43.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:44.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:44.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:45.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:45.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:46.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:46.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:47.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:47.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:48.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:48.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:49.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:49.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:50.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:50.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:51.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:51.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:52.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:52.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:53.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:53.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:54.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:54.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:55.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:55.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:56.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:56.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:57.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:57.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:58.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:58.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:02:59.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:02:59.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:00.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:00.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:01.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:01.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:02.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:02.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:03.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:03.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:04.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:04.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:05.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:05.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:06.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:06.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:07.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:07.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:08.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:08.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:09.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:09.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:10.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:10.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:11.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:11.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:12.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:12.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:13.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:13.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:14.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:14.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:15.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:15.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:16.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:16.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:17.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:17.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:18.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:18.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:19.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:19.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:20.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:20.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:21.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:21.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:22.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:22.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:23.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:23.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:24.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:24.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:25.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:25.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:26.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:26.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:27.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:27.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:28.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:28.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:29.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:29.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:30.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:30.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:31.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:31.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:32.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:32.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:33.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:33.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:34.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:34.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:35.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:35.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:36.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:36.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:37.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:37.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:38.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:38.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:39.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:39.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:40.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:40.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:41.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:41.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:42.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:42.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:43.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:43.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:44.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:44.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:45.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:45.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:46.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:46.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:47.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:47.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:48.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:48.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:49.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:49.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:49.580 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:03:49.581 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:03:49.581 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:03:49.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:03:49.582 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:03:49.588 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:03:49.588 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:03:49.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:03:49.588 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:03:50.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:50.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:50.558 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1422,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:03:50.580 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1422,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:03:51.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:51.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:52.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:52.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:53.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:53.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:53.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:03:53.581 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:03:54.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:54.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:55.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:55.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:56.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:56.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:57.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:57.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:58.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:58.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:03:59.091 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:03:59.121 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:03:59.121 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:03:59.122 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:03:59.122 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:03:59.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:03:59.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:00.130 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1428,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:00.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:00.161 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:04:00.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:00.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:01.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:01.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:01.387 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1428,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:01.417 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:01.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:04:02.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:02.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:02.423 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1428,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:02.453 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:04:02.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:02.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:04:03.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:03.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:03.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:04:04.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:04.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:04.427 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:04:04.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:04.427 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:04:05.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:05.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:05.436 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:05.468 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:05.468 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:04:05.469 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:04:05.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:05.469 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:04:06.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:06.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:06.470 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1428,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:06.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:06.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:04:07.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:07.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:07.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1428,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:07.426 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:07.426 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:04:08.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:08.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:08.412 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1428,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:08.442 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:04:08.442 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:04:08.443 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:04:08.443 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:04:08.443 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:04:08.443 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:04:08.460 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:04:08.460 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:04:08.461 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:08.461 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:04:09.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:09.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:09.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:09.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:10.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:10.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:11.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:11.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:12.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:12.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:12.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:12.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:04:13.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:13.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:13.503 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:13.533 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:14.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:14.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:15.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:15.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:16.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:16.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:16.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:16.533 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:04:17.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:17.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:17.498 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:17.521 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1428,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:18.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:18.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:19.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:19.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:20.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:20.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:20.522 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:20.522 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:04:21.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:21.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:22.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:22.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:23.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:23.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:23.621 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1432,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:23.633 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1432,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:04:23.633 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:04:23.633 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:23.634 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:04:24.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:24.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:24.602 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1432,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:24.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:04:24.627 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:04:25.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:25.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:25.597 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1432,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:04:25.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:04:26.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:26.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:27.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:27.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:28.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:28.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:29.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:29.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:30.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:30.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:31.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:31.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:32.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:32.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:33.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:33.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:34.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:34.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:35.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:35.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:36.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:36.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:37.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:37.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:38.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:38.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:39.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:39.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:40.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:40.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:41.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:41.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:42.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:42.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:43.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:43.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:44.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:44.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:45.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:45.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:46.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:46.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:47.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:47.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:48.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:48.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:49.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:49.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:50.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:50.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:51.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:51.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:52.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:52.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:53.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:53.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:54.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:54.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:55.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:55.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:56.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:56.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:57.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:57.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:58.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:58.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:04:59.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:04:59.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:00.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:00.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:01.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:01.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:02.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:02.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:03.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:03.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:04.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:04.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:05.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:05.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:06.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:06.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:07.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:07.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:08.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:08.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:09.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:09.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:10.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:10.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:11.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:11.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 105 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:12.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:12.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:13.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:13.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:14.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:14.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:15.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:15.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:16.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:16.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:17.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:17.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:18.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:18.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:19.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:19.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:20.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:20.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:21.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:21.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:21.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:21.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:22.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:22.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:24.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:24.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:25.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:25.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:25.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:25.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:26.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:26.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:27.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:27.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:28.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:28.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:29.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:29.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:30.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:30.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:31.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:31.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:32.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:32.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:33.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:33.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:34.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:34.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:35.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:35.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:36.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:36.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:37.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:37.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:38.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:38.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:39.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:39.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:40.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:40.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:41.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:41.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:42.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:42.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:43.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:43.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:44.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:44.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:45.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:45.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:46.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:46.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:47.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:47.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:48.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:48.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:49.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:49.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:50.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:50.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:51.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:51.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:52.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:52.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:53.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:53.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:54.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:54.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:55.621 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:05:55.621 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:05:55.621 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:05:55.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:05:55.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:05:55.627 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:05:55.628 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:05:55.628 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:05:55.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:05:55.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:55.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:56.629 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1432,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:05:56.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1432,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:05:56.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:56.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:57.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:57.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:58.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:58.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:05:59.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:05:59.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:05:59.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:05:59.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:00.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:00.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:01.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:01.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:02.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:02.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:03.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:03.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:04.627 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:04.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:04.650 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:06:04.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:04.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:06:04.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:04.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:05.624 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1438,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:05.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:05.654 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:06:05.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:05.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:06.670 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1438,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:06.692 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:06.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:06:06.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:06.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:07.722 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1438,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:07.752 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:06:07.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:07.753 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:06:07.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:07.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:08.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:08.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:08.876 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:06:09.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:09.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:09.878 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:06:09.878 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:09.878 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:06:10.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:10.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:10.871 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:10.892 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:10.893 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:06:10.893 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:06:10.893 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:10.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:06:11.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:11.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:11.877 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1438,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:11.904 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:11.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:06:12.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:12.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:12.928 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1438,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:12.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:12.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:06:13.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:13.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:13.902 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1438,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:13.924 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:06:13.925 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:06:13.925 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:06:13.925 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:06:13.925 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:06:13.925 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:06:13.976 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:06:13.976 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:06:13.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:13.977 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:06:14.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:14.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:14.935 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:14.957 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:15.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:15.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:16.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:16.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:17.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:17.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:17.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:17.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:06:18.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:18.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:18.967 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:18.989 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:19.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:19.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:20.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:20.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:21.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:21.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:21.990 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:21.990 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:06:22.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:22.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:22.984 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:23.015 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1438,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:23.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:23.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:24.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:24.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:25.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:25.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:26.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:26.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:06:26.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:26.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:27.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:27.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:28.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:28.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:29.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:29.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:30.072 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:30.102 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:06:30.103 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:06:30.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:30.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:06:30.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:30.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:31.128 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1442,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:31.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:06:31.159 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:06:31.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:31.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:32.165 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1442,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:06:32.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:06:32.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:32.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:33.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:33.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:34.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:34.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:35.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:35.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:36.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:36.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:37.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:37.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:38.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:38.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:39.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:39.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:40.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:40.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:41.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:41.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:42.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:42.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:43.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:43.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:44.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:44.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:45.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:45.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:46.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:46.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:47.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:47.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:48.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:48.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:49.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:49.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:50.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:50.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:51.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:51.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:52.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:52.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:53.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:53.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:54.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:54.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:55.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:55.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:56.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:56.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:57.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:57.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:58.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:58.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:06:59.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:06:59.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:00.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:00.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:01.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:01.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:02.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:02.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:03.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:03.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:04.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:04.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:05.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:05.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:06.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:06.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:07.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:07.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:08.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:08.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:09.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:09.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:10.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:10.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:11.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:11.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:12.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:12.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:13.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:13.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:14.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:14.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:15.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:15.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:16.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:16.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:17.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:17.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:18.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:18.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:19.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:19.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:20.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:20.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:21.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:21.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:22.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:22.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:23.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:23.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:24.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:24.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:25.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:25.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:26.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:26.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:27.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:27.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:28.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:28.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:29.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:29.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:30.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:30.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:31.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:31.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:32.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:32.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:33.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:33.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:34.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:34.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:35.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:35.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:36.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:36.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:37.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:37.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:38.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:38.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:39.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:39.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:40.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:40.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:41.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:41.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:42.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:42.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:43.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:43.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:44.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:44.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:45.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:45.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:46.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:46.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:47.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:47.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:48.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:48.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:49.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:49.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:50.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:50.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:51.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:51.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:52.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:52.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:53.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:53.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:54.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:54.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:55.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:55.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:56.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:56.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:57.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:57.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:58.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:58.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:07:59.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:07:59.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:00.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:00.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:01.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:01.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:02.198 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:08:02.199 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:08:02.199 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:08:02.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:02.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:08:02.205 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:08:02.205 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:08:02.205 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:02.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:02.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:02.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:03.168 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:03.201 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:03.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:03.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:04.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:04.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:05.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:05.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:06.202 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:06.202 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:06.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:06.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:07.190 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:07.213 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1442,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:07.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:07.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:08.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:08.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:09.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:09.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:10.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:10.214 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:10.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:10.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:11.241 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:11.264 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:11.264 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:08:11.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:11.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:08:11.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:11.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:12.275 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1444,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:12.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:12.298 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:08:12.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:12.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:13.291 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1444,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:13.321 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:13.321 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:08:13.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:13.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:14.322 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1444,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:14.345 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:08:14.345 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:14.345 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:08:14.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:14.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:15.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:15.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:15.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:08:16.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:16.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:16.403 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:08:16.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:16.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:17.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:17.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:17.377 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:17.408 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:17.408 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:08:17.408 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:08:17.408 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:17.408 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:08:18.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:18.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:18.411 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1444,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:18.434 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:18.434 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:08:19.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:19.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:19.424 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1444,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:19.445 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:19.445 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:08:20.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:20.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:20.445 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1444,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:20.466 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:08:20.467 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:08:20.467 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:08:20.467 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:08:20.467 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:08:20.467 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:08:20.480 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:08:20.480 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:08:20.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:20.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:21.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:21.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:21.532 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:21.562 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:22.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:22.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:23.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:23.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:24.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:24.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:24.564 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:24.564 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:25.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:25.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:25.557 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:25.574 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:26.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:26.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:27.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:27.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:28.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:28.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:28.575 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:28.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:29.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:29.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:29.586 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:29.608 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1444,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:30.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:30.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:31.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:31.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:32.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:32.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:32.609 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:32.609 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:08:33.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:33.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:34.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:34.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:35.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:35.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:35.446 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:35.479 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:08:35.479 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:08:35.479 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:35.479 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:08:36.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:36.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:36.472 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1448,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:36.496 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:08:36.497 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:08:37.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:37.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:37.499 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1448,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:08:37.522 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:08:38.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:38.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:39.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:39.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:40.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:40.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:41.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:41.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:42.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:42.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:43.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:43.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:44.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:44.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:45.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:45.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:46.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:46.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:47.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:47.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:48.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:48.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:49.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:49.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:50.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:50.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:51.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:51.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:52.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:52.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:53.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:53.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:54.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:54.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:55.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:55.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:56.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:56.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:57.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:57.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:58.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:58.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:08:59.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:08:59.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:00.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:00.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:01.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:01.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:02.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:02.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:03.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:03.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:04.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:04.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:05.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:05.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:06.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:06.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:07.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:07.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:08.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:08.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:09.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:09.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:10.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:10.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:11.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:11.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:12.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:12.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:13.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:13.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:14.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:14.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:15.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:15.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:16.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:16.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:17.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:17.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:18.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:18.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:19.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:19.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:20.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:20.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:21.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:21.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:22.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:22.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:23.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:23.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:24.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:24.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:25.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:25.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:26.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:26.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:27.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:27.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:28.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:28.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:29.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:29.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:30.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:30.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:31.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:31.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:32.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:32.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:33.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:33.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:34.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:34.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:35.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:35.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:36.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:36.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:37.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:37.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:38.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:38.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:39.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:39.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:40.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:40.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:41.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:41.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:42.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:42.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:43.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:43.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:44.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:44.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:45.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:45.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:46.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:46.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:47.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:47.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:48.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:48.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:49.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:49.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:50.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:50.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:51.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:51.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:52.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:52.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:53.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:53.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:54.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:54.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:55.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:55.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:56.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:56.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:57.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:57.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:58.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:58.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:09:59.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:09:59.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:00.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:00.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:01.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:01.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:02.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:02.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:03.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:03.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:04.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:04.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:05.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:05.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:06.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:06.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:07.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:07.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:07.523 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:10:07.524 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:10:07.524 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:10:07.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:07.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:10:07.531 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:10:07.531 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:10:07.531 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:07.531 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:08.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:08.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:08.547 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:08.570 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:09.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:09.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:10.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:10.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:11.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:11.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:11.572 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:11.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:12.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:12.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:12.570 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:12.600 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1448,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:13.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:13.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:14.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:14.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:15.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:15.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:15.601 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:15.602 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:16.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:16.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:17.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:17.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:18.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:18.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:19.036 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1452,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:19.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1452,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:19.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:19.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:20.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:20.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:21.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:21.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:22.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:22.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:22.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:22.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:23.037 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:23.069 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:23.069 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:10:23.070 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:23.070 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:10:23.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:23.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:24.054 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1454,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:24.085 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:24.085 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:10:24.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:24.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:25.303 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1454,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:25.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:25.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:10:25.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:25.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:26.328 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1454,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:26.350 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:10:26.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:26.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:10:26.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:26.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:27.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:27.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:27.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:10:28.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:28.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:28.481 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:10:28.482 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:28.482 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:29.472 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:29.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:29.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:29.493 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:29.493 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:10:29.493 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:10:29.494 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:29.494 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:10:30.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:30.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:30.517 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1454,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:30.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:30.541 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:10:31.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:31.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:31.540 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1454,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:31.563 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:31.563 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:10:32.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:32.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:32.536 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1454,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:32.558 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:10:32.558 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:10:32.558 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:10:32.558 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:10:32.559 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:10:32.559 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:10:32.576 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:10:32.576 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:10:32.576 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:32.576 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:33.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:33.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:33.627 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:33.645 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:34.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:34.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:35.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:35.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:36.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:36.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:36.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:36.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:37.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:37.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:37.667 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:37.689 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1454,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:38.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:38.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:39.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:39.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:40.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:40.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:40.690 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:40.690 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:10:41.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:41.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:42.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:42.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:43.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:43.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:44.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:44.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:45.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:45.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:46.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:46.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:46.729 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1458,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:46.761 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1458,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:10:46.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:10:46.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:46.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:10:47.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:47.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:47.771 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1458,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:47.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:10:47.804 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:10:48.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:48.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:48.808 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1458,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:10:48.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:10:49.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:49.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:50.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:50.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:51.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:51.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:52.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:52.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:53.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:53.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:54.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:54.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:55.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:55.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:56.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:56.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:57.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:57.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:58.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:58.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:10:59.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:10:59.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:00.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:00.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:01.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:01.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:02.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:02.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:03.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:03.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:04.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:04.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:05.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:05.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:06.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:06.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:07.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:07.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:08.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:08.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:09.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:09.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:10.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:10.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:11.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:11.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:12.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:12.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:13.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:13.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:14.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:14.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:15.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:15.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:16.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:16.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:17.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:17.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:18.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:18.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:19.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:19.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:20.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:20.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:21.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:21.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:22.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:22.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:23.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:23.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:24.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:24.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:25.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:25.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:26.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:26.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:27.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:27.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:28.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:28.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:29.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:29.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:30.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:30.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:31.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:31.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:32.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:32.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:33.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:33.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:34.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:34.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:35.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:35.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:36.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:36.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:37.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:37.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:38.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:38.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:39.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:39.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:40.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:40.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:41.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:41.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:42.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:42.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:43.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:43.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:44.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:44.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:45.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:45.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:46.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:46.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:47.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:47.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:48.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:48.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:49.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:49.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:50.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:50.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:51.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:51.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:52.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:52.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:53.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:53.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:54.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:54.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:55.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:55.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:56.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:56.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:57.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:57.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:58.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:58.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:11:59.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:11:59.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:00.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:00.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:01.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:01.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:02.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:02.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:03.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:03.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:04.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:04.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:05.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:05.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:06.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:06.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:07.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:07.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:08.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:08.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:09.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:09.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:10.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:10.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:11.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:11.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:12.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:12.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:13.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:13.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:14.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:14.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:15.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:15.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:16.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:16.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:17.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:17.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:18.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:18.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:18.830 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:12:18.830 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:12:18.831 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:12:18.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:18.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:12:18.838 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:12:18.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:12:18.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:18.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:19.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:19.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:19.821 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1458,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:19.851 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1458,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:20.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:20.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:21.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:21.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:22.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:22.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:22.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:22.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:23.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:23.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:24.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:24.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:25.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:25.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:26.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:26.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:26.735 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1462,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:26.756 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1462,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:27.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:27.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:28.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:28.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:29.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:29.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:29.757 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:29.757 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:30.363 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T08:12:30.351+0000 7f0f56ffd700 0 --2- 172.21.6.124:0/2417118782 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f0f5808bf40 0x7f0f5808c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T08:12:30.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:30.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:30.993 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:31.023 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:31.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:12:31.024 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:31.024 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:12:31.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:31.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:32.014 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1464,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:32.036 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:32.036 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:12:32.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:32.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:33.054 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1464,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:33.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:33.087 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:12:33.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:33.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:34.105 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1464,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:34.135 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:12:34.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:34.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:12:34.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:34.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:35.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:12:35.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:35.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:36.243 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:12:36.243 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:36.244 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:36.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:36.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:37.256 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:37.288 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:37.288 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:12:37.288 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:12:37.288 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:37.288 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:12:37.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:37.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:38.318 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1464,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:38.347 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:38.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:12:38.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:38.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:39.351 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1464,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:39.373 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:39.373 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:12:39.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:39.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:40.374 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1464,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:12:40.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:12:40.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:12:40.405 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:12:40.406 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:12:40.406 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:12:40.406 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:12:40.426 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:12:40.426 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:12:40.426 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:40.426 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:40.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:40.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:41.440 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:41.441 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:41.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:41.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:42.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:42.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:43.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:43.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:44.441 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:44.442 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:44.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:44.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:45.443 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:45.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:45.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:45.476 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1464,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:12:46.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:46.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:47.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:47.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:48.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:12:48.477 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:12:48.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:48.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:49.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:49.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:50.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:50.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:51.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:51.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:52.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:52.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:53.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:53.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:54.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:54.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:55.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:55.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:56.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:56.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:57.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:57.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:58.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:58.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:12:59.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:12:59.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:00.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:00.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:00.905 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1468,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:13:00.937 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1468,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:13:00.938 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:13:00.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:13:00.938 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:13:01.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:01.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:01.941 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1468,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:13:01.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:13:01.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:13:02.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:02.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:02.996 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1468,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:13:03.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:13:03.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:03.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:04.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:04.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:05.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:05.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:06.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:06.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:07.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:07.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:08.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:08.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:09.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:09.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:10.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:10.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:11.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:11.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:12.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:12.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:13.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:13.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:14.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:14.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:15.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:15.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:16.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:16.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:17.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:17.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:18.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:18.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:19.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:19.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:20.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:20.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:21.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:21.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:22.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:22.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:23.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:23.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:24.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:24.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:25.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:25.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:26.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:26.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:27.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:27.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:28.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:28.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:29.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:29.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:30.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:30.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:31.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:31.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:32.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:32.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:33.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:33.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:34.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:34.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:35.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:35.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:36.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:36.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:37.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:37.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:38.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:38.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:39.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:39.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:40.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:40.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:41.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:41.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:42.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:42.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:43.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:43.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:44.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:44.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:45.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:45.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:46.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:46.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:47.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:47.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:48.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:48.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:49.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:49.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:50.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:50.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:51.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:51.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:52.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:52.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:53.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:53.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:54.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:54.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:55.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:55.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:56.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:56.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:57.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:57.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:58.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:58.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:13:59.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:13:59.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:00.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:00.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:01.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:01.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:02.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:02.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:03.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:03.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:04.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:04.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:05.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:05.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:06.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:06.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:07.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:07.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:08.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:08.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:09.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:09.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:10.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:10.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:11.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:11.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:12.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:12.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:13.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:13.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:14.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:14.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:15.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:15.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:16.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:16.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:17.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:17.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:18.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:18.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:19.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:19.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:20.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:20.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:21.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:21.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:22.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:22.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:23.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:23.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:24.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:24.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:25.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:25.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:26.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:26.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:27.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:27.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:28.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:28.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:29.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:29.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:30.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:30.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:31.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:31.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:32.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:32.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:33.019 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:14:33.020 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:14:33.020 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:14:33.020 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:14:33.020 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:14:33.026 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:14:33.027 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:14:33.027 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:33.027 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:33.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:33.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:34.003 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1468,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:34.033 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1468,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:34.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:34.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:35.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:35.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:36.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:36.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:37.034 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:37.034 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:37.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:37.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:38.033 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:38.055 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:38.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:14:38.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:38.056 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:14:38.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:38.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:39.068 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1472,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:14:39.100 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:39.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:14:39.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:39.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:40.106 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1472,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:14:40.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:40.138 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:14:40.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:40.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:41.154 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1472,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:14:41.183 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:14:41.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:41.183 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:14:41.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:41.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:42.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:14:42.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:42.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:43.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:14:43.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:43.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:43.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:43.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:44.295 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:44.326 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:44.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:14:44.327 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:14:44.327 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:44.327 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:14:44.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:44.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:45.322 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1472,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:14:45.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:45.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:14:45.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:45.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:46.337 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1472,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:14:46.367 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:46.367 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:14:46.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:46.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:47.368 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1472,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:14:47.390 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:14:47.390 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:14:47.390 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:14:47.391 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:14:47.391 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:14:47.391 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:14:47.410 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:14:47.411 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:14:47.411 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:47.411 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:47.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:47.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:48.485 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:48.516 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:48.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:48.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:49.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:49.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:50.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:50.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:51.517 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:51.518 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:51.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:51.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:52.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:52.561 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1472,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:14:52.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:52.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:53.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:53.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:54.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:54.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:55.563 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:14:55.563 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:14:55.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:55.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:56.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:56.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:57.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:57.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:58.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:58.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:14:59.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:14:59.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:00.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:00.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:01.447 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:15:01.480 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:15:01.480 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:15:01.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:15:01.480 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:15:01.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:01.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:02.479 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1476,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:15:02.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:15:02.510 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:15:02.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:02.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:03.508 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1476,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:15:03.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:15:03.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:03.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:04.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:04.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:05.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:05.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:06.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:06.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:07.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:07.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:08.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:08.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:09.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:09.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:10.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:10.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:11.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:11.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:12.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:12.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:13.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:13.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:14.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:14.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:15.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:15.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:16.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:16.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:17.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:17.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:18.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:18.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:19.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:19.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:20.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:20.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:21.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:21.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:22.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:22.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:23.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:23.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:24.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:24.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:25.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:25.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:26.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:26.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:27.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:27.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:28.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:28.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:29.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:29.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:30.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:30.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:31.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:31.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:32.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:32.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:33.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:33.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:34.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:34.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:35.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:35.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:36.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:36.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:37.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:37.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:38.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:38.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:39.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:39.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:40.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:40.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:41.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:41.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:42.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:42.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:43.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:43.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:44.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:44.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:45.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:45.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:46.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:46.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:47.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:47.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:48.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:48.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:49.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:49.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:50.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:50.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:51.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:51.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:52.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:52.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:53.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:53.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:54.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:54.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:55.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:55.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:56.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:56.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:57.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:57.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:58.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:58.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:15:59.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:15:59.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:00.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:00.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:01.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:01.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:02.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:02.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:03.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:03.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:04.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:04.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:05.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:05.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:06.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:06.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:07.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:07.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:08.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:08.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:09.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:09.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:10.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:10.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:11.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:11.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:12.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:12.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:13.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:13.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:14.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:14.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:15.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:15.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:16.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:16.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:17.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:17.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:18.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:18.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:19.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:19.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:20.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:20.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:21.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:21.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:22.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:22.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:23.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:23.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:24.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:24.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:25.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:25.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:26.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:26.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:27.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:27.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:28.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:28.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:29.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:29.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:30.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:30.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:31.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:31.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:32.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:32.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:33.530 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:16:33.530 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:16:33.530 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:16:33.531 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:33.531 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:16:33.536 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:16:33.536 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:16:33.537 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:33.537 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:16:33.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:33.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:34.523 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:34.552 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:34.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:34.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:35.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:35.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:36.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:36.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:37.553 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:37.553 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:16:37.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:37.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:38.565 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:38.585 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1476,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:38.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:38.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:39.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:39.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:40.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:40.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:41.586 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:41.587 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:16:41.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:41.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:42.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:42.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:43.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:43.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:44.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:44.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:45.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:45.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:46.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:46.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:47.604 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1480,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:47.627 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1480,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:47.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:47.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:48.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:48.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:49.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:49.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:50.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:50.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:16:50.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:50.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:51.605 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:51.637 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:51.637 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:16:51.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:51.638 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:16:52.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:52.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:52.643 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1482,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:16:52.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:52.673 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:16:52.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:52.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:53.699 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1482,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:16:53.728 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:53.728 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:16:53.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:53.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:54.730 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1482,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:16:54.751 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:16:54.751 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:54.751 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:16:55.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:55.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:55.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:16:55.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:55.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:56.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:16:56.849 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:56.849 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:16:56.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:56.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:57.864 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:57.893 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:16:57.894 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:16:57.894 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:16:57.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:57.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:16:57.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:57.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:58.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:58.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:58.908 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1482,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:16:58.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:16:58.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:16:59.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:16:59.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:16:59.992 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1482,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:17:00.020 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:00.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:17:00.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:00.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:01.032 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1482,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:17:01.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:17:01.055 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:17:01.055 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:17:01.056 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:17:01.056 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:17:01.056 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:17:01.088 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:17:01.088 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:17:01.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:01.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:17:01.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:01.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:02.165 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:02.187 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:02.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:02.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:03.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:03.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:04.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:04.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:05.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:05.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:17:05.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:05.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:06.162 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:06.184 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:06.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:06.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:07.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:07.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:08.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:08.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:09.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:09.186 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:17:09.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:09.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:10.175 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:10.199 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1482,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:10.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:10.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:11.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:11.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:12.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:12.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:13.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:13.201 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:17:13.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:13.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:14.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:14.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:15.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:15.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:16.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:16.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:17.246 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:17.267 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:17:17.268 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:17:17.268 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:17.268 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:17:17.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:17.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:18.257 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1486,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:17:18.287 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:17:18.287 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:17:18.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:18.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:19.293 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1486,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:17:19.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:17:19.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:19.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:20.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:20.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:21.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:21.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:22.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:22.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:23.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:23.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:24.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:24.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:25.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:25.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:26.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:26.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:27.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:27.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:28.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:28.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:29.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:29.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:30.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:30.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:31.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:31.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:32.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:32.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:33.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:33.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:34.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:34.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:35.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:35.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:36.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:36.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:37.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:37.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:38.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:38.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:39.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:39.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:40.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:40.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:41.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:41.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:42.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:42.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:43.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:43.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:44.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:44.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:45.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:45.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:46.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:46.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:47.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:47.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:48.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:48.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:49.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:49.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:50.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:50.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:51.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:51.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:52.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:52.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:53.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:53.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:54.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:54.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:55.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:55.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:56.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:56.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:57.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:57.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:58.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:58.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:17:59.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:17:59.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:01.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:01.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:02.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:02.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:03.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:03.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:03.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:03.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:04.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:04.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:05.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:05.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:06.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:06.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:07.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:07.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:08.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:08.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:09.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:09.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:10.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:10.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:11.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:11.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:12.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:12.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:13.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:13.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:14.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:14.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:15.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:15.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:16.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:16.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:17.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:17.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:18.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:18.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:19.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:19.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:20.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:20.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:21.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:21.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:22.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:22.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:23.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:23.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:24.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:24.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:25.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:25.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:26.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:26.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:27.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:27.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:28.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:28.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:29.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:29.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:30.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:30.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:31.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:31.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:32.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:32.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:33.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:33.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:34.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:34.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:35.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:35.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:36.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:36.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:37.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:37.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:38.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:38.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:39.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:39.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:40.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:40.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:41.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:41.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:42.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:42.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:43.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:43.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:44.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:44.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:45.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:45.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:46.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:46.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:47.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:47.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:48.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:48.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:49.326 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:18:49.383 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:18:49.383 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:18:49.384 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:18:49.384 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:18:49.412 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:18:49.412 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:18:49.412 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:18:49.413 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:18:49.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:49.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:50.418 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:18:50.440 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:18:50.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:50.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:51.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:51.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:52.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:52.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:53.440 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:18:53.441 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:18:53.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:53.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:54.447 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:18:54.470 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1486,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:18:54.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:54.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:55.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:55.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:56.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:56.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:57.471 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:18:57.472 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:18:57.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:57.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:58.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:58.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:18:59.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:18:59.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:00.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:00.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:01.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:01.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:02.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:02.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:03.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:03.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:04.669 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:04.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:04.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:04.699 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:04.699 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:19:04.699 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:04.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:19:05.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:05.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:05.699 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1494,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:05.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:05.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:19:06.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1494,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:06.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:06.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:06.718 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:06.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:19:07.670 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1494,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:07.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:07.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:07.699 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:19:07.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:07.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:19:08.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:08.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:08.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:19:09.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:09.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:09.801 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:19:09.802 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:09.802 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:19:10.783 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:10.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:10.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:10.815 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1494,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:10.815 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:19:10.815 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:19:10.815 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:10.815 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:19:11.789 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1494,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:11.811 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:11.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:19:11.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:11.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:12.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:12.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:12.805 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1494,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:12.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:12.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:19:13.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:13.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:13.938 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1494,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:13.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:19:13.960 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T08:19:13.960 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:19:13.960 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:19:13.960 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:19:13.961 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:19:13.985 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:19:13.985 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:19:13.985 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:19:13.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:19:14.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:14.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:15.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:15.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:16.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:16.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:17.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:17.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:18.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:18.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:19.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:19.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:20.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:20.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:21.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:21.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:22.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:22.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:23.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:23.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:24.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:24.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:25.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:25.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:26.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:26.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:27.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:27.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:28.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:28.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:28.986 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:19:28.986 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:19:28.987 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:19:28.987 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:28.987 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:19:29.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:29.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:30.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:30.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:31.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:31.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:32.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:32.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:33.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:33.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:34.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:34.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:35.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:35.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:36.209 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1496,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:36.241 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1496,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:19:36.241 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:19:36.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:36.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:19:36.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:36.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:37.241 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1496,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:19:37.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:19:37.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:19:37.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:37.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:38.268 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1496,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:19:38.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:19:38.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:38.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:39.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:39.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:40.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:40.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:41.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:41.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:43.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:42.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:43.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:43.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:44.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:44.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:45.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:45.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:46.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:46.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:48.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:48.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:49.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:49.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:49.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:49.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:50.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:50.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:52.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:52.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:53.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:53.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:54.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:54.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:55.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:55.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:55.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:55.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:57.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:57.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:58.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:58.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:58.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:58.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:19:59.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:19:59.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:00.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:00.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:02.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:02.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:03.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:03.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:04.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:04.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:05.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:05.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:06.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:06.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:07.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:07.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:08.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:08.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:09.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:09.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:10.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:10.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:11.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:11.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:12.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:12.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:13.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:13.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:14.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:14.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:15.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:15.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:16.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:16.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:17.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:17.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:18.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:18.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:19.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:19.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:20.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:20.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:21.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:21.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:22.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:22.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:23.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:23.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:24.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:24.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:25.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:25.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:26.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:26.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:27.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:27.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:28.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:28.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:29.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:29.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:30.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:30.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:31.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:31.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:32.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:32.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:33.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:33.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:34.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:34.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:35.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:35.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:36.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:36.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:37.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:37.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:38.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:38.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:39.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:39.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:40.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:40.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:41.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:41.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:42.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:42.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:43.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:43.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:44.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:44.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:45.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:45.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:46.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:46.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:47.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:47.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:48.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:48.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:49.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:49.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:50.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:50.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:51.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:51.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:52.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:52.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:53.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:53.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:54.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:54.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:55.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:55.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:56.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:56.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:57.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:57.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:58.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:58.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:20:59.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:20:59.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:00.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:00.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:01.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:01.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:02.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:02.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:03.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:03.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:04.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:04.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:05.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:05.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:06.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:06.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:07.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:07.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:08.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:08.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:08.301 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:21:08.301 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:21:08.302 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:21:08.302 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:21:08.302 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:21:08.309 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:21:08.309 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:21:08.309 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:21:08.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:21:09.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:09.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:10.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:10.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:11.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:11.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:12.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:12.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:13.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:13.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:14.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:14.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:15.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:15.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:16.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:16.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:17.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:17.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:18.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:18.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:19.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:19.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:20.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:20.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:21.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:21.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:22.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:22.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:23.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:23.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:23.310 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:21:23.310 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:21:23.310 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:21:23.311 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:23.311 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:21:24.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:24.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:24.326 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:24.349 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:24.349 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:21:24.349 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:24.350 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:21:25.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:25.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:25.346 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1504,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:21:25.377 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:25.377 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:21:26.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:26.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:26.351 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1504,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:21:26.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:26.370 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:21:27.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:27.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:27.355 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1504,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:21:27.378 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:21:27.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:27.378 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:21:28.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:28.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:28.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:21:29.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:29.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:29.488 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:21:29.488 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:29.489 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:21:30.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:30.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:30.485 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:30.507 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:30.507 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:21:30.507 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:21:30.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:30.508 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:21:31.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:31.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:31.509 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1504,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:21:31.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:31.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:21:32.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:32.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:32.548 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1504,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:21:32.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:32.581 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:21:33.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:33.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:33.588 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1504,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:21:33.610 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:21:33.611 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:21:33.611 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:21:33.611 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:21:33.611 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:21:33.612 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:21:33.634 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:21:33.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:21:33.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:33.635 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:21:34.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:34.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:34.701 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:34.728 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1504,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:35.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:35.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:36.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:36.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:37.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:37.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:37.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:37.730 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:21:38.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:38.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:39.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:39.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:40.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:40.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:41.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:41.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:42.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:42.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:43.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:43.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:44.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:44.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:45.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:45.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:46.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:46.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:47.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:47.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:48.448 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1508,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:48.469 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1508,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:21:48.470 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:21:48.470 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:48.470 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:21:48.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:48.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:49.480 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1508,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:21:49.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:21:49.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:21:49.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:49.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:50.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:50.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:50.515 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1508,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:21:50.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:21:51.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:51.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:52.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:52.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:53.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:53.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:54.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:54.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:55.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:55.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:56.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:56.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:57.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:57.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:58.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:58.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:21:59.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:21:59.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:00.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:00.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:01.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:01.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:02.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:02.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:03.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:03.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:04.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:04.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:05.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:05.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:06.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:06.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:07.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:07.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:08.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:08.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:09.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:09.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:10.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:10.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:11.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:11.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:12.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:12.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:13.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:13.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:14.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:14.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:15.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:15.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:16.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:16.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:17.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:17.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:18.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:18.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:19.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:19.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:20.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:20.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:21.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:21.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:22.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:22.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:23.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:23.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:24.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:24.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:25.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:25.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:26.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:26.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:27.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:27.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:28.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:28.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:29.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:29.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:30.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:30.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:31.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:31.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:32.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:32.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:33.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:33.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:34.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:34.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:35.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:35.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:36.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:36.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:37.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:37.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:38.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:38.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:39.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:39.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:40.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:40.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:41.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:41.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:42.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:42.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:43.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:43.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:44.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:44.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:45.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:45.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:46.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:46.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:47.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:47.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:48.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:48.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:49.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:49.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:50.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:50.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:51.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:51.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:52.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:52.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:53.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:53.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:54.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:54.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:55.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:55.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:56.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:56.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:57.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:57.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:58.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:58.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:22:59.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:22:59.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:00.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:00.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:01.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:01.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:02.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:02.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:03.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:03.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:04.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:04.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:05.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:05.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:06.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:06.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:07.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:07.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:08.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:08.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:09.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:09.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:10.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:10.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:11.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:11.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:12.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:12.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:13.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:13.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:14.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:14.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:15.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:15.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:16.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:16.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:17.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:17.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:18.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:18.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:19.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:19.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:20.539 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:23:20.540 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:23:20.540 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:23:20.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:20.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:23:20.547 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:23:20.547 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:23:20.548 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:20.548 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:20.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:20.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:21.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1508,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:21.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:21.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:21.561 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1508,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:22.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:22.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:23.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:23.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:24.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:24.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:24.562 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:24.562 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:25.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:25.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:26.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:26.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:27.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:27.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:28.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:28.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:29.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:29.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:30.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:30.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:31.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:31.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:31.765 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:31.785 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:31.786 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:23:31.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:31.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:23:32.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:32.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:32.684 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1514,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:23:32.716 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:32.716 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:23:33.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:33.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:33.767 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1514,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:23:33.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:33.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:23:34.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:34.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:34.824 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1514,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:23:34.853 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:23:34.854 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:34.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:23:35.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:35.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:35.943 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:23:36.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:36.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:36.944 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:23:36.944 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:36.945 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:37.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:37.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:37.960 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:37.991 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:37.992 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:23:37.992 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:23:37.992 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:37.992 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:23:38.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:38.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:39.059 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1514,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:23:39.081 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:39.081 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:23:39.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:39.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:40.090 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1514,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:23:40.122 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:40.122 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:23:40.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:40.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:41.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1514,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:23:41.137 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:23:41.137 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:23:41.137 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:23:41.138 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:23:41.138 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:23:41.138 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:23:41.157 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:23:41.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:23:41.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:41.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:41.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:41.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:42.200 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:42.223 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:42.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:42.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:43.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:43.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:44.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:44.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:45.224 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:45.224 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:45.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:45.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:46.222 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:46.254 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:46.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:46.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:47.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:47.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:48.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:48.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:49.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:49.256 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:49.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:49.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:50.281 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:50.312 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1514,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:50.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:50.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:51.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:51.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:52.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:52.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:53.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:53.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:23:53.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:53.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:54.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:54.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:55.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:55.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:56.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:56.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:56.725 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1518,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:56.747 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1518,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:23:56.747 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:23:56.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:56.747 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:23:57.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:57.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:57.732 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1518,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:23:57.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:23:57.762 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:23:58.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:58.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:23:58.775 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1518,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:23:58.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:23:59.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:23:59.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:00.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:00.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:01.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:01.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:02.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:02.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:03.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:03.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:04.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:04.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:05.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:05.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:06.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:06.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:07.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:07.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:08.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:08.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:09.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:09.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:10.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:10.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:11.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:11.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:12.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:12.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:13.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:13.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:14.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:14.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:15.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:15.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:16.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:16.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:17.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:17.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:18.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:18.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:19.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:19.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:20.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:20.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:21.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:21.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:22.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:22.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:23.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:23.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:24.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:24.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:25.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:25.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:26.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:26.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:27.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:27.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:28.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:28.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:29.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:29.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:30.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:30.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:31.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:31.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:32.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:32.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:33.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:33.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:34.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:34.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:35.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:35.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:36.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:36.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:37.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:37.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:38.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:38.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:39.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:39.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:40.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:40.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:41.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:41.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:42.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:42.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:43.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:43.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:44.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:44.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:45.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:45.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:46.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:46.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:47.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:47.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:48.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:48.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:49.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:49.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:50.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:50.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:51.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:51.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:52.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:52.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:53.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:53.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:54.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:54.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:55.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:55.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:56.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:56.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:57.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:57.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:58.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:58.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:24:59.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:24:59.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:00.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:00.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:01.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:01.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:02.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:02.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:03.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:03.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:04.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:04.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:05.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:05.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:06.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:06.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:07.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:07.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:08.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:08.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:09.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:09.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:10.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:10.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:11.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:11.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:12.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:12.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:13.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:13.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:14.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:14.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:15.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:15.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:16.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:16.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:17.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:17.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:18.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:18.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:19.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:19.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:20.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:20.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:21.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:21.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:22.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:22.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:23.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:23.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:24.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:24.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:25.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:25.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:26.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:26.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:27.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:27.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:28.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:28.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:28.799 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:25:28.799 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:25:28.800 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:25:28.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:28.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:25:28.806 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:25:28.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:25:28.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:28.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:29.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:29.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:29.780 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1518,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:29.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1518,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:30.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:30.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:31.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:31.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:32.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:32.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:32.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:32.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:33.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:33.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:34.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:34.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:35.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:35.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:36.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:36.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:37.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:37.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:37.614 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:37.646 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:37.646 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:25:37.647 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:37.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:25:38.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:38.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:38.641 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1524,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:38.673 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:38.673 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:25:39.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:39.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:39.886 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1524,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:39.908 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:39.908 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:25:40.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:40.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:40.908 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1524,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:40.938 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:25:40.938 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:40.938 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:25:41.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:41.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:42.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:25:42.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:42.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:43.059 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:25:43.060 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:43.060 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:43.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:43.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:44.065 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:44.097 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:44.097 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:25:44.097 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:25:44.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:44.098 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:25:44.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:44.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:45.064 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1524,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:45.092 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:45.092 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:25:45.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:45.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:46.096 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1524,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:46.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:46.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:25:46.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:46.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:47.156 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1524,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:25:47.186 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:25:47.186 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:25:47.187 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:25:47.187 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:25:47.187 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:25:47.187 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:25:47.218 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:25:47.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:25:47.218 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:47.218 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:47.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:47.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:48.262 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:48.283 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:48.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:48.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:49.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:49.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:50.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:50.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:51.284 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:51.285 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:51.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:51.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:52.279 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:52.309 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:52.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:52.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:53.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:53.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:54.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:54.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:55.310 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:55.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:25:55.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:55.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:56.403 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:56.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:56.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:56.434 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1524,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:25:57.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:57.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:58.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:58.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:59.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:25:59.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:25:59.436 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:25:59.436 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:26:00.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:00.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:01.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:01.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:02.390 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:26:02.422 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:26:02.422 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:26:02.423 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:26:02.423 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:26:02.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:02.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:03.413 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1528,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:26:03.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:26:03.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:26:03.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:03.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:04.502 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1528,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:26:04.521 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:26:04.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:04.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:05.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:05.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:06.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:06.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:07.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:07.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:08.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:08.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:09.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:09.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:10.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:10.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:11.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:11.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:12.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:12.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:13.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:13.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:14.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:14.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:15.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:15.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:16.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:16.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:17.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:17.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:18.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:18.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:19.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:19.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:20.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:20.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:21.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:21.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:22.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:22.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:23.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:23.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:24.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:24.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:25.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:25.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:26.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:26.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:27.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:27.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:28.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:28.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:29.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:29.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:30.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:30.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:31.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:31.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:32.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:32.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:33.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:33.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:34.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:34.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:35.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:35.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:36.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:36.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:37.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:37.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:38.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:38.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:39.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:39.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:40.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:40.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:41.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:41.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:42.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:42.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:43.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:43.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:44.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:44.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:45.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:45.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:46.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:46.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:47.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:47.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:48.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:48.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:49.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:49.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:50.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:50.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:51.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:51.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:52.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:52.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:53.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:53.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:54.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:54.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:55.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:55.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:56.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:56.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:57.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:57.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:58.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:58.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:26:59.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:26:59.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:00.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:00.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:01.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:01.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:02.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:02.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:03.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:03.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:04.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:04.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:05.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:05.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:06.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:06.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:07.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:07.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:08.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:08.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:09.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:09.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:10.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:10.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:11.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:11.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:12.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:12.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:13.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:13.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:14.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:14.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:15.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:15.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:16.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:16.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:17.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:17.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:18.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:18.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:19.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:19.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:20.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:20.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:21.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:21.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:22.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:22.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:23.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:23.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:24.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:24.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:25.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:25.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:26.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:26.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:27.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:27.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:28.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:28.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:29.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:29.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:30.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:30.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:31.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:31.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:32.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:32.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:33.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:33.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:34.523 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:27:34.523 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:27:34.523 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:27:34.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:34.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:27:34.530 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:27:34.530 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:27:34.530 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:34.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:34.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:34.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:35.507 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:35.528 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:35.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:35.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:36.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:36.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:37.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:37.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:38.529 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:38.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:38.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:38.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:39.527 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:39.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:39.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:39.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:40.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:40.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:41.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:41.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:42.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:42.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:42.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:42.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:43.591 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:43.605 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1528,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:43.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:43.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:44.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:44.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:45.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:45.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:46.606 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:46.606 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:46.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:46.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:47.600 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:47.641 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:47.641 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:27:47.642 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:47.642 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:27:47.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:47.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:48.625 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1530,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:27:48.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:48.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:48.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:48.657 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:27:49.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:49.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:49.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1530,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:27:49.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:49.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:27:50.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:50.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:50.747 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1530,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:27:50.770 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:27:50.770 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:50.770 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:27:51.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:51.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:51.818 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:27:52.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:52.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:52.820 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:27:52.820 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:52.820 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:53.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:53.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:53.805 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:53.827 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:53.828 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:27:53.828 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:27:53.828 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:53.828 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:27:54.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:54.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:54.736 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1530,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:27:54.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:54.767 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:27:55.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:55.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:55.730 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1530,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:27:55.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:55.753 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:27:56.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:56.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:56.758 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1530,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:27:56.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:27:56.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:27:56.778 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:27:56.779 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:27:56.779 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:27:56.779 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:27:56.806 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:27:56.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:27:56.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:27:56.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:27:57.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:57.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:57.753 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:57.785 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:27:58.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:58.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:27:59.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:27:59.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:00.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:00.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:00.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:28:00.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:28:01.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:01.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:01.761 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:28:01.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1530,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:28:02.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:02.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:03.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:03.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:04.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:04.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:04.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:28:04.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:28:05.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:05.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:06.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:06.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:07.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:07.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:08.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:08.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:09.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:09.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:10.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:10.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:11.443 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:28:11.475 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:28:11.476 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:28:11.476 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:28:11.476 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:28:11.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:11.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:12.488 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1532,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:28:12.511 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:28:12.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:28:12.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:12.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:13.529 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1532,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:28:13.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:28:13.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:13.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:14.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:14.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:15.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:15.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:16.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:16.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:17.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:17.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:18.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:18.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:19.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:19.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:20.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:20.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:21.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:21.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:22.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:22.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:23.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:23.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:24.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:24.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:25.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:25.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:26.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:26.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:27.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:27.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:28.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:28.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:29.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:29.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:30.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:30.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:31.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:31.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:32.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:32.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:33.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:33.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:34.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:34.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:35.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:35.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:36.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:36.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:37.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:37.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:38.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:38.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:39.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:39.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:40.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:40.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:41.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:41.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:42.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:42.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:43.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:43.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:44.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:44.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:45.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:45.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:46.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:46.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:47.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:47.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:48.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:48.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:49.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:49.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:50.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:50.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:51.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:51.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:52.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:52.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:53.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:53.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:54.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:54.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:55.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:55.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:56.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:56.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:57.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:57.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:58.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:58.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:28:59.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:28:59.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:00.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:00.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:01.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:01.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:02.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:02.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:03.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:03.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:04.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:04.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:05.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:05.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:06.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:06.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:07.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:07.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:08.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:08.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:09.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:09.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:10.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:10.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:11.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:11.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:12.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:12.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:13.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:13.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:14.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:14.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:15.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:15.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:16.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:16.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:17.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:17.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:18.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:18.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:19.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:19.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:20.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:20.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:21.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:21.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:22.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:22.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:23.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:23.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:24.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:24.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:25.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:25.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:26.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:26.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:27.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:27.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:28.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:28.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:29.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:29.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:30.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:30.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:31.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:31.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:32.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:32.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:33.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:33.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:34.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:34.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:35.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:35.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:36.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:36.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:37.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:37.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:38.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:38.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:39.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:39.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:40.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:40.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:41.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:41.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:42.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:42.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:43.562 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:29:43.562 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:29:43.563 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:29:43.563 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:29:43.563 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:29:43.569 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:29:43.570 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:29:43.570 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:29:43.570 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:29:43.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:43.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:44.548 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:44.572 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:44.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:44.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:45.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:45.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:46.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:46.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:47.573 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:29:47.573 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:29:47.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:47.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:48.568 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:48.599 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:48.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:48.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:49.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:49.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:50.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:50.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:51.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:29:51.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:29:51.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:51.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:52.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:52.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:52.595 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:52.627 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:53.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:53.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:54.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:54.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:55.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:55.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:55.628 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:29:55.628 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:29:56.597 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:56.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:56.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:56.629 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1532,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:29:57.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:57.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:58.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:58.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:59.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:29:59.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:29:59.630 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:29:59.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:30:00.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:00.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:00.625 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:00.648 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:00.648 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:30:00.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:00.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:30:01.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:01.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:01.649 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1536,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:30:01.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:01.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:30:02.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:02.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:02.657 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1536,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}]}} 2019-10-01T08:30:02.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:02.688 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:30:03.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:03.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:03.652 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1536,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:30:03.683 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:30:03.683 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:03.683 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:30:04.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:04.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:04.760 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:30:05.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:05.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:05.761 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:30:05.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:05.762 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:30:06.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:06.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:06.756 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:06.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:06.777 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:30:06.777 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:30:06.777 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:06.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:30:07.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:07.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:07.776 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1536,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:30:07.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:07.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:30:08.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:08.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:08.785 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1536,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}]}} 2019-10-01T08:30:08.815 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:08.816 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:30:09.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:09.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:09.767 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1536,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:30:09.784 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:30:09.784 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:30:09.785 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:30:09.785 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:30:09.785 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:30:09.786 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:30:09.807 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:30:09.807 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:30:09.808 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:09.808 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:30:10.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:10.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:10.804 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:10.835 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:11.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:11.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:12.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:12.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:13.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:13.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:13.836 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:13.836 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:30:14.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:14.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:14.852 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:14.874 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1536,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:15.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:15.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:16.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:16.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:17.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:17.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:17.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:17.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:30:18.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:18.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:19.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:19.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:20.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:20.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:21.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:21.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:22.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:22.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:23.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:23.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:24.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:24.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:24.639 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:24.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:30:24.670 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:30:24.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:24.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:30:25.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:25.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:25.662 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1538,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:30:25.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:30:25.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:30:26.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:26.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:26.669 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1538,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}]}} 2019-10-01T08:30:26.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:30:27.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:27.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:28.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:28.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:29.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:29.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:30.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:30.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:31.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:31.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:32.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:32.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:33.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:33.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:34.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:34.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:35.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:35.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:36.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:36.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:37.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:37.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:38.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:38.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:39.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:39.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:40.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:40.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:41.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:41.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:42.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:42.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:43.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:43.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:44.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:44.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:45.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:45.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:46.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:46.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:47.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:47.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:48.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:48.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:49.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:49.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:50.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:50.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:51.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:51.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:52.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:52.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:53.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:53.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:54.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:54.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:55.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:55.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:56.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:56.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:57.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:57.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:58.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:58.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:30:59.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:30:59.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:00.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:00.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:01.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:01.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:02.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:02.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:03.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:03.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:04.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:04.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:05.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:05.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:06.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:06.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:07.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:07.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:08.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:08.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:09.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:09.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:10.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:10.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:11.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:11.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:12.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:12.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:13.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:13.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:14.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:14.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:15.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:15.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:16.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:16.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:17.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:17.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:18.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:18.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:19.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:19.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:20.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:20.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:21.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:21.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:22.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:22.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:23.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:23.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:24.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:24.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:25.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:25.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:26.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:26.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:27.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:27.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:28.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:28.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:29.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:29.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:30.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:30.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:31.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:31.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:32.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:32.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:33.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:33.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:34.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:34.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:35.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:35.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:36.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:36.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:37.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:37.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:38.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:38.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:39.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:39.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:40.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:40.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:41.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:41.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:42.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:42.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:43.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:43.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:44.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:44.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:45.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:45.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:46.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:46.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:47.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:47.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:48.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:48.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:49.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:49.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:50.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:50.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:51.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:51.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:52.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:52.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:53.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:53.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:54.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:54.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:55.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:55.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:56.692 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:31:56.692 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:31:56.692 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:31:56.693 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:31:56.693 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:31:56.699 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:31:56.699 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:31:56.699 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:31:56.699 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:31:56.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:56.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:57.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:57.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:57.807 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:31:57.836 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:31:58.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:58.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:31:59.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:31:59.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:00.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:00.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:00.837 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:00.837 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:01.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:01.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:01.819 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:01.846 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:02.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:02.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:03.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:03.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:04.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:04.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:04.848 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:04.848 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:05.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:05.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:05.841 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:05.862 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:06.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:06.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:07.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:07.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:08.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:08.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:08.863 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:08.863 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:09.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:09.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:09.855 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:09.874 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1538,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:10.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:10.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:11.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:11.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:12.875 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:12.875 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:12.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:12.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:13.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:13.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:14.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:14.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:15.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:15.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:15.938 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1542,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:15.959 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1542,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:16.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:16.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:17.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:17.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:18.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:18.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:18.961 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:18.961 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:19.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:19.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:20.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:20.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:21.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:21.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:21.869 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:21.891 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1546,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:22.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:22.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:23.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:23.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:24.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:24.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:24.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:24.893 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:25.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:25.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:25.935 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:25.967 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:25.967 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:32:25.967 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:25.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:32:26.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:26.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:26.980 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1550,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:32:27.003 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:27.003 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:32:27.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:27.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:28.003 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1550,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}]}} 2019-10-01T08:32:28.027 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:28.027 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:32:28.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:28.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:29.017 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1550,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:32:29.048 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:32:29.048 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:29.048 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:32:29.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:29.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:30.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:32:30.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:30.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:31.142 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:32:31.143 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:31.143 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:31.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:31.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:32.121 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:32.150 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:32.150 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:32:32.150 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:32:32.151 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:32.151 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:32:32.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:32.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:33.135 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1550,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:32:33.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:33.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:32:33.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:33.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:34.159 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1550,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}]}} 2019-10-01T08:32:34.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:34.190 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:32:34.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:34.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:35.142 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1550,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:32:35.164 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:32:35.164 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:32:35.165 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:32:35.165 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:32:35.165 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:32:35.165 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:32:35.180 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:32:35.180 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:32:35.180 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:35.180 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:35.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:35.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:36.235 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:36.257 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:36.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:36.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:37.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:37.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:38.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:38.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:39.258 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:39.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:39.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:39.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:40.271 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:40.301 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1550,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:40.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:40.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:41.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:41.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:42.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:42.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:43.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:43.302 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:32:43.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:43.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:44.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:44.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:45.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:45.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:46.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:46.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:47.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:47.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:48.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:48.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:49.225 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1554,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:49.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1554,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:32:49.259 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:32:49.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:49.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:32:49.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:49.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:50.258 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1554,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:32:50.280 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:32:50.281 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:32:50.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:50.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:51.279 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1554,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:32:51.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:32:51.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:51.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:52.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:52.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:53.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:53.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:54.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:54.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:55.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:55.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:56.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:56.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:57.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:57.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:58.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:58.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:32:59.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:32:59.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:00.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:00.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:01.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:01.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:02.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:02.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:03.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:03.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:04.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:04.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:05.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:05.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:06.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:06.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:07.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:07.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:08.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:08.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:09.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:09.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:10.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:10.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:11.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:11.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:12.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:12.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:13.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:13.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:14.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:14.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:15.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:15.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:16.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:16.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:17.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:17.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:18.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:18.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:19.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:19.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:20.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:20.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:21.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:21.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:22.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:22.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:23.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:23.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:24.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:24.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:25.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:25.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:26.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:26.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:27.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:27.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:28.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:28.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:29.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:29.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:30.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:30.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:31.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:31.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:32.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:32.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:33.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:33.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:34.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:34.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:35.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:35.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:36.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:36.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:37.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:37.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:38.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:38.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:39.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:39.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:40.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:40.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:41.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:41.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:42.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:42.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:43.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:43.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:44.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:44.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:45.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:45.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:46.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:46.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:47.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:47.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:48.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:48.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:49.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:49.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:50.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:50.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:51.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:51.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:52.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:52.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:53.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:53.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:54.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:54.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:55.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:55.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:56.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:56.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:57.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:57.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:58.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:58.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:33:59.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:33:59.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:00.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:00.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:01.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:01.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:02.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:02.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:03.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:03.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:04.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:04.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:05.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:05.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:06.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:06.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:07.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:07.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:08.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:08.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:09.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:09.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:10.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:10.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:11.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:11.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:12.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:12.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:13.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:13.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:14.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:14.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:15.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:15.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:16.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:16.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:17.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:17.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:18.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:18.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:19.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:19.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:20.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:20.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:21.310 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:34:21.310 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:34:21.311 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:34:21.311 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:21.311 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:34:21.317 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:34:21.318 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:34:21.318 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:21.318 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:21.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:21.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:22.329 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1554,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:22.352 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1554,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:22.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:22.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:23.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:23.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:24.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:24.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:25.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:25.356 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:25.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:25.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:26.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:26.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:27.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:27.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:28.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:28.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:29.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:29.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:30.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:30.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:30.866 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:30.889 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:30.890 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:34:30.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:30.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:34:31.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:31.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:31.913 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1560,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:31.944 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:31.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:34:32.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:32.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:32.923 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1560,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:32.945 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:32.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:34:33.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:33.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:33.851 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1560,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:33.873 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:34:33.874 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:33.874 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:34:34.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:34.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:34.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:34:35.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:35.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:35.979 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:34:35.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:35.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:36.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:36.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:36.988 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:37.009 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:37.009 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:34:37.009 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:34:37.010 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:37.010 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:34:37.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:37.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:37.983 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1560,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:38.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:38.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:34:38.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:38.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:39.005 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1560,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:39.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:39.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:34:39.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:39.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:40.009 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1560,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:40.031 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:34:40.031 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:34:40.031 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:34:40.031 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:34:40.032 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:34:40.032 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:34:40.048 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:34:40.048 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:34:40.048 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:40.049 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:40.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:40.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:41.104 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:41.135 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:41.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:41.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:42.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:42.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:43.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:43.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:44.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:44.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:44.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:44.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:45.152 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:45.173 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1560,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:45.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:45.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:46.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:46.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:47.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:47.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:48.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:48.175 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:34:48.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:48.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:49.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:49.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:50.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:50.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:51.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:51.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:52.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:52.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:53.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:53.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:54.122 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1564,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:54.153 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1564,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:34:54.154 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:34:54.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:54.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:34:54.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:54.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:55.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1564,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:55.202 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:34:55.203 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:34:55.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:55.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:56.173 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1564,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:34:56.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:34:56.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:56.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:57.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:57.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:58.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:58.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:34:59.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:34:59.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:00.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:00.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:01.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:01.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:02.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:02.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:03.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:03.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:04.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:04.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:05.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:05.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:06.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:06.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:07.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:07.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:08.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:08.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:09.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:09.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:10.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:10.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:11.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:11.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:12.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:12.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:13.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:13.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:14.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:14.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:15.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:15.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:16.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:16.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:17.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:17.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:18.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:18.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:19.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:19.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:20.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:20.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:21.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:21.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:22.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:22.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:23.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:23.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:24.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:24.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:25.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:25.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:26.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:26.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:27.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:27.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:28.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:28.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:29.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:29.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:30.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:30.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:31.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:31.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:32.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:32.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:33.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:33.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:34.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:34.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:35.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:35.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:36.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:36.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:37.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:37.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:38.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:38.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:39.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:39.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:40.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:40.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:41.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:41.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:42.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:42.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:43.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:43.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:44.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:44.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:45.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:45.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:46.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:46.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:47.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:47.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:48.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:48.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:49.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:49.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:50.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:50.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:51.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:51.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:53.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:53.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:53.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:53.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:54.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:54.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:55.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:55.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:56.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:56.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:57.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:57.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:58.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:58.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:35:59.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:35:59.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:00.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:00.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:02.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:02.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:02.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:02.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:03.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:03.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:05.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:05.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:06.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:06.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:07.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:07.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:07.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:07.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:09.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:09.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:10.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:10.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:11.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:11.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:12.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:12.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:13.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:13.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:14.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:14.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:15.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:15.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:16.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:16.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:17.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:17.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:18.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:18.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:19.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:19.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:20.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:20.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:21.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:21.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:22.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:22.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:23.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:23.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:24.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:24.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:25.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:25.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:26.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:26.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:26.197 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:36:26.197 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:36:26.198 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:36:26.198 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:26.198 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:36:26.204 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:36:26.204 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:36:26.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:26.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:27.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:27.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:27.167 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1564,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:27.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1564,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:28.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:28.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:29.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:29.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:30.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:30.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:30.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:30.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:31.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:31.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:32.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:32.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:33.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:33.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:34.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:34.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:35.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:35.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:35.768 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:35.790 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:35.791 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:36:35.791 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:35.791 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:36:36.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:36.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:36.776 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1570,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:36.798 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:36.798 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:36:37.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:37.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:38.030 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1570,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:38.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:38.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:36:38.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:38.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:39.050 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1570,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:39.073 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:36:39.073 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:39.073 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:36:39.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:39.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:40.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:40.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:40.269 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:36:41.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:41.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:41.269 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:36:41.270 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:41.270 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:42.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:42.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:42.283 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:42.309 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:42.310 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:36:42.310 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:36:42.310 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:42.310 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:36:43.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:43.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:43.294 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1570,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:43.317 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:43.317 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:36:44.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:44.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:44.324 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1570,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:44.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:44.346 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:36:45.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:45.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:45.341 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1570,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:36:45.370 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:36:45.371 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:36:45.371 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:36:45.371 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:36:45.371 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:36:45.372 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:36:45.396 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:36:45.396 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:36:45.397 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:45.397 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:46.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:46.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:46.410 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:46.442 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:47.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:47.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:48.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:48.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:49.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:49.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:49.443 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:49.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:50.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:50.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:50.452 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:50.475 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:51.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:51.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:52.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:52.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:53.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:53.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:53.476 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:53.476 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:54.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:54.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:54.490 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:54.513 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1570,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:36:55.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:55.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:56.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:56.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:57.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:57.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:57.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:36:57.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:36:58.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:58.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:36:59.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:36:59.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:00.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:00.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:01.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:01.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:01.573 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:37:01.597 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:37:01.597 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:37:01.598 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:37:01.598 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:37:02.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:02.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:02.597 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1572,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:37:02.619 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:37:02.619 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:37:03.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:03.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:03.613 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1572,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:37:03.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:37:04.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:04.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:05.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:05.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:06.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:06.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:07.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:07.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:08.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:08.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:09.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:09.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:10.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:10.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:11.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:11.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:12.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:12.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:13.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:13.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:14.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:14.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:15.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:15.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:16.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:16.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:17.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:17.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:18.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:18.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:19.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:19.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:19.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:19.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:20.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:20.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:21.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:21.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:23.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:22.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:23.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:23.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:24.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:24.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:25.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:25.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:26.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:26.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:27.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:27.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:28.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:28.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:29.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:29.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:30.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:30.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:31.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:31.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:32.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:32.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:33.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:33.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:34.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:34.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:35.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:35.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:37.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:37.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:38.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:38.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:39.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:39.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:40.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:40.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:41.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:41.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:42.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:42.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:43.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:43.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:44.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:44.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:45.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:45.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:46.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:46.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:47.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:47.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:48.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:48.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:49.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:49.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:50.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:50.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:51.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:51.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:52.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:52.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:53.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:53.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:54.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:54.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:55.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:55.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:56.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:56.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:57.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:57.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:58.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:58.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:37:59.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:37:59.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:00.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:00.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:01.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:01.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:02.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:02.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:03.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:03.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:04.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:04.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:05.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:05.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:06.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:06.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:07.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:07.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:08.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:08.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:09.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:09.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:10.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:10.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:11.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:11.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:12.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:12.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:13.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:13.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:14.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:14.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:15.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:15.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:16.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:16.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:17.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:17.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:18.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:18.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:19.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:19.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:20.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:20.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:21.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:21.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:22.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:22.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:23.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:23.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:24.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:24.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:25.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:25.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:26.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:26.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:27.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:27.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:28.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:28.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:29.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:29.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:30.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:30.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:31.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:31.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:32.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:32.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:33.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:33.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:33.643 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:38:33.643 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:38:33.643 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:38:33.644 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:38:33.644 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:38:33.654 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:38:33.654 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:38:33.655 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:33.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:34.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:34.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:34.665 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:34.688 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:35.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:35.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:36.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:36.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:37.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:37.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:37.689 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:37.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:38.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:38.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:38.661 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:38.693 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:39.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:39.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:40.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:40.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:41.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:41.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:41.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:41.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:42.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:42.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:42.694 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:42.726 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:43.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:43.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:44.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:44.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:45.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:45.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:45.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:45.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:46.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:46.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:46.754 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:46.779 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1572,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:47.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:47.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:48.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:48.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:49.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:49.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:49.779 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:49.779 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:50.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:50.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:50.802 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1576,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:50.824 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1576,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:50.824 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:38:50.824 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:50.824 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:38:51.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:51.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:51.768 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1576,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:51.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:51.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:38:52.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:52.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:52.808 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1576,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:52.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:52.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:38:53.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:53.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:53.848 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1576,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:53.878 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:38:53.879 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:53.879 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:38:54.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:54.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:54.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:38:55.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:55.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:55.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:38:55.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:55.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:38:56.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:56.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:56.934 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1576,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:56.966 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1576,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:38:56.966 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:38:56.966 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:38:56.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:56.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:38:57.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:57.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:57.937 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1576,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:57.968 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:57.968 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:38:58.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:58.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:58.894 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1576,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:58.926 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:38:58.926 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:38:59.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:38:59.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:38:59.932 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1576,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:38:59.955 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:38:59.955 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T08:38:59.955 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:38:59.956 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:38:59.956 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:38:59.956 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:38:59.974 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:38:59.974 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:38:59.975 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:38:59.975 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:39:00.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:00.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:01.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:01.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:02.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:02.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:03.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:03.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:04.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:04.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:05.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:05.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:06.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:06.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:07.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:07.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:08.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:08.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:09.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:09.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:10.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:10.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:11.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:11.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:12.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:12.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:13.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:13.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:14.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:14.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:14.976 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:39:14.977 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:39:14.977 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:39:14.977 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:39:14.978 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:39:15.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:15.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:16.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:16.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:17.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:17.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:18.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:18.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:19.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:19.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:20.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:20.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:21.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:21.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:22.221 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1578,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:39:22.242 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1578,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:39:22.242 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:39:22.243 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:39:22.243 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:39:22.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:22.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:23.237 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1578,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:39:23.267 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:39:23.267 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:39:23.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:23.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:24.264 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1578,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:39:24.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:39:24.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:24.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:25.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:25.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:26.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:26.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:27.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:27.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:28.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:28.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:29.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:29.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:30.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:30.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:31.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:31.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:32.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:32.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:33.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:33.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:34.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:34.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:35.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:35.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:36.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:36.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:37.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:37.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:38.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:38.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:39.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:39.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:40.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:40.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:41.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:41.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:42.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:42.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:43.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:43.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:44.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:44.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:45.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:45.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:46.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:46.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:47.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:47.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:48.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:48.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:49.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:49.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:50.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:50.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:51.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:51.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:52.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:52.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:53.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:53.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:54.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:54.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:55.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:55.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:56.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:56.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:57.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:57.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:58.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:58.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:39:59.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:39:59.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:00.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:00.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:01.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:01.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:02.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:02.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:03.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:03.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:04.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:04.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:05.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:05.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:06.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:06.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:07.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:07.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:08.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:08.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:09.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:09.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:10.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:10.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:11.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:11.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:12.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:12.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:13.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:13.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:14.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:14.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:15.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:15.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:16.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:16.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:17.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:17.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:18.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:18.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:19.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:19.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:20.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:20.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:21.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:21.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:22.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:22.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:23.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:23.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:24.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:24.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:25.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:25.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:26.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:26.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:27.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:27.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:28.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:28.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:29.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:29.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:30.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:30.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:31.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:31.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:32.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:32.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:33.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:33.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:34.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:34.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:35.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:35.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:36.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:36.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:37.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:37.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:38.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:38.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:39.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:39.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:40.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:40.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:41.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:41.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:42.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:42.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:43.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:43.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:44.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:44.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:45.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:45.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:46.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:46.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:47.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:47.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:48.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:48.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:49.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:49.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:50.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:50.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:51.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:51.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:52.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:52.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:53.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:53.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:54.295 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:40:54.295 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:40:54.295 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:40:54.295 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:40:54.295 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:40:54.303 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:40:54.303 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:40:54.304 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:40:54.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:40:54.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:54.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:55.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:55.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:56.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:56.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:57.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:57.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:58.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:58.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:40:59.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:40:59.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:00.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:00.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:01.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:01.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:02.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:02.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:03.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:03.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:04.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:04.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:05.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:05.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:06.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:06.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:07.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:07.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:08.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:08.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:09.304 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:41:09.305 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:41:09.305 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:41:09.306 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:09.306 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:09.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:09.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:10.326 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:10.352 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:10.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:41:10.353 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:10.353 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:41:10.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:10.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:11.377 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1586,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:11.407 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:11.407 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:41:11.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:11.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:12.421 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1586,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:12.442 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:12.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:41:12.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:12.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:13.440 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1586,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:13.463 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:41:13.463 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:13.463 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:41:13.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:13.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:14.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:14.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:14.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:41:15.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:15.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:15.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:41:15.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:15.636 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:16.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:16.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:16.643 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:16.668 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:16.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:41:16.668 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:41:16.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:16.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:41:17.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:17.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:17.666 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1586,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:17.689 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:17.689 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:41:18.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:18.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:18.664 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1586,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:18.686 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:18.687 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:41:19.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:19.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:19.700 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1586,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:19.723 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:41:19.723 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:41:19.723 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:41:19.723 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:41:19.724 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:41:19.724 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:41:19.756 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:41:19.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:41:19.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:19.757 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:20.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:20.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:20.798 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:20.821 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:21.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:21.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:22.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:22.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:23.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:23.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:23.822 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:23.822 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:24.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:24.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:24.811 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:24.834 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:25.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:25.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:26.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:26.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:27.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:27.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:27.834 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:27.834 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:28.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:28.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:28.858 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:28.880 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1586,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:29.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:29.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:30.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:30.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:31.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:31.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:31.881 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:31.881 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:41:32.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:32.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:33.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:33.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:34.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:34.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:35.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:35.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:35.930 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1590,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:35.961 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1590,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:41:35.961 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:41:35.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:35.962 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:41:36.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:36.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:36.956 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1590,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:41:36.988 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:41:36.988 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:41:37.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:37.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:37.970 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1590,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:41:37.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:41:38.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:38.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:39.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:39.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:40.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:40.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:41.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:41.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:42.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:42.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:43.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:43.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:44.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:44.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:45.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:45.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:46.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:46.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:47.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:47.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:48.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:48.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:49.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:49.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:50.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:50.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:51.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:51.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:52.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:52.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:53.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:53.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:54.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:54.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:55.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:55.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:56.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:56.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:57.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:57.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:58.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:58.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:41:59.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:41:59.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:00.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:00.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:01.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:01.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:02.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:02.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:03.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:03.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:04.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:04.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:05.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:05.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:06.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:06.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:07.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:07.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:08.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:08.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:09.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:09.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:10.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:10.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:11.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:11.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:12.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:12.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:13.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:13.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:14.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:14.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:15.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:15.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:16.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:16.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:17.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:17.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:18.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:18.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:19.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:19.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:20.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:20.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:21.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:21.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:22.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:22.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:23.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:23.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:24.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:24.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:25.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:25.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:26.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:26.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:27.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:27.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:28.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:28.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:29.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:29.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:30.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:30.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:31.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:31.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:32.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:32.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:33.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:33.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:34.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:34.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:35.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:35.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:36.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:36.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:37.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:37.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:38.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:38.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:39.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:39.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:40.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:40.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:41.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:41.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:42.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:42.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:43.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:43.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:44.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:44.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:45.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:45.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:46.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:46.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:47.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:47.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:48.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:48.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:49.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:49.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:50.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:50.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:51.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:51.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:52.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:52.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:53.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:53.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:54.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:54.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:55.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:55.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:56.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:56.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:57.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:57.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:58.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:58.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:42:59.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:42:59.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:00.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:00.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:01.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:01.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:02.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:02.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:03.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:03.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:04.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:04.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:05.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:05.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:06.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:06.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:07.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:07.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:07.993 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:43:07.993 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:43:07.993 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:43:07.993 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:07.993 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:43:07.999 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:43:07.999 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:43:07.999 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:07.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:08.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:08.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:09.001 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1590,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:09.033 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1590,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:09.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:09.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:10.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:10.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:11.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:11.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:12.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:12.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:12.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:12.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:13.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:13.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:14.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:14.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:15.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:15.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:16.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:16.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:17.541 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:17.572 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:17.572 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:43:17.572 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:17.572 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:43:17.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:17.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:18.565 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1598,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:43:18.587 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:18.587 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:43:18.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:18.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:19.574 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1598,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:43:19.596 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:19.596 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:43:19.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:19.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:20.610 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1598,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:43:20.633 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:43:20.633 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:20.633 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:43:20.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:20.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:21.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:21.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:21.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:43:22.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:22.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:22.710 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:43:22.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:22.711 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:23.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:23.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:23.693 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:23.723 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:23.724 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:43:23.724 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:43:23.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:23.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:43:24.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:24.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:24.726 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1598,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:43:24.758 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:24.758 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:43:25.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:25.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:25.813 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1598,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:43:25.840 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:25.840 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:43:26.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:26.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:26.854 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1598,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:43:26.884 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:43:26.884 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:43:26.884 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:43:26.885 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:43:26.885 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:43:26.885 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:43:26.905 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:43:26.905 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:43:26.906 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:26.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:27.526 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T08:43:27.523+0000 7f3ac97ca700 0 --2- 172.21.6.124:0/832299033 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f3aa8006010 0x7f3aa8006400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T08:43:27.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:27.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:28.099 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:28.122 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:28.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:28.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:29.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:29.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:30.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:30.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:31.122 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:31.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:31.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:31.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:32.141 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:32.171 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1598,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:32.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:32.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:33.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:33.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:34.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:34.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:35.172 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:35.172 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:43:35.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:35.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:36.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:36.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:37.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:37.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:38.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:38.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:39.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:39.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:40.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:40.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:41.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:41.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:42.466 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1600,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:42.498 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1600,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:43:42.498 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:43:42.498 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:42.498 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:43:42.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:42.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:43.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1600,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:43:43.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:43:43.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:43:43.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:43.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:44.567 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1600,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:43:44.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:43:44.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:44.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:45.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:45.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:46.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:46.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:47.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:47.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:48.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:48.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:49.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:49.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:50.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:50.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:51.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:51.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:52.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:52.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:53.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:53.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:54.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:54.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:55.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:55.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:56.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:56.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:57.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:57.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:58.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:58.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:43:59.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:43:59.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:00.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:00.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:01.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:01.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:02.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:02.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:03.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:03.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:04.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:04.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:05.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:05.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:06.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:06.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:07.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:07.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:08.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:08.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:09.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:09.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:10.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:10.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:11.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:11.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:12.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:12.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:13.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:13.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:14.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:14.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:15.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:15.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:16.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:16.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:17.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:17.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:18.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:18.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:19.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:19.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:20.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:20.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:21.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:21.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:22.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:22.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:23.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:23.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:24.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:24.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:25.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:25.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:26.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:26.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:27.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:27.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:28.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:28.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:29.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:29.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:30.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:30.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:31.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:31.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:32.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:32.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:33.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:33.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:34.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:34.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:35.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:35.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:36.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:36.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:37.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:37.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:38.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:38.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:39.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:39.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:40.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:40.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:41.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:41.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:42.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:42.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:43.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:43.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:44.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:44.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:45.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:45.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:46.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:46.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:47.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:47.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:48.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:48.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:49.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:49.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:50.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:50.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:51.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:51.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:52.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:52.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:53.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:53.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:54.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:54.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:55.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:55.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:56.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:56.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:57.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:57.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:58.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:58.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:44:59.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:44:59.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:00.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:00.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:01.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:01.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:02.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:02.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:03.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:03.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:04.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:04.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:05.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:05.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:06.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:06.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:07.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:07.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:08.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:08.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:09.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:09.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:10.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:10.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:11.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:11.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:12.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:12.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:13.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:13.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:14.590 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:45:14.590 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:45:14.590 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:45:14.591 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:45:14.591 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:45:14.597 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:45:14.597 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:45:14.597 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:14.597 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:45:14.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:14.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:15.628 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1600,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:15.660 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1600,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:15.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:15.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:16.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:16.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:17.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:17.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:18.661 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:18.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:45:18.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:18.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:19.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:19.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:20.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:20.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:21.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:21.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:22.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:22.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:23.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:23.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:24.267 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:24.297 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1604,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:24.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:24.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:25.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:25.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:26.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:26.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:27.298 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:27.299 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:45:27.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:27.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:28.284 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1608,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:28.303 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1608,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:28.304 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:45:28.304 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:28.304 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:45:28.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:28.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:29.273 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1608,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:45:29.305 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:29.305 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:45:29.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:29.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:30.315 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1608,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:45:30.346 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:30.346 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:45:30.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:30.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:31.350 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1608,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:45:31.371 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:45:31.371 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:31.371 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:45:31.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:31.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:32.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:45:32.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:32.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:33.492 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:45:33.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:33.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:45:33.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:33.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:34.495 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1608,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:34.525 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1608,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:34.525 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:45:34.525 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:45:34.525 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:34.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:45:34.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:34.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:35.511 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1608,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:45:35.543 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:35.543 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:45:35.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:35.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:36.545 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1608,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:45:36.565 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:36.565 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:45:36.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:36.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:37.568 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1608,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:45:37.590 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:45:37.590 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T08:45:37.590 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:45:37.590 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:45:37.590 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:45:37.591 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:45:37.609 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:45:37.609 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:45:37.609 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:45:37.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:45:37.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:37.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:38.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:38.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:39.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:39.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:40.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:40.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:41.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:41.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:42.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:42.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:43.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:43.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:44.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:44.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:45.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:45.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:46.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:46.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:47.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:47.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:48.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:48.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:49.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:49.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:50.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:50.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:51.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:51.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:52.610 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:45:52.610 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:45:52.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:45:52.611 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:52.611 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:45:52.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:52.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:53.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:53.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:54.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:54.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:55.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:55.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:56.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:56.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:57.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:57.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:58.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:58.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:59.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:45:59.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:45:59.885 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:59.907 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1610,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:45:59.907 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:45:59.907 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:45:59.907 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:46:00.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:00.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:00.909 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1610,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:46:00.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:46:00.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:46:01.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:01.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:01.863 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1610,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:46:01.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:46:02.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:02.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:03.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:03.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:04.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:04.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:05.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:05.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:06.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:06.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:07.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:07.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:08.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:08.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:09.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:09.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:10.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:10.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:11.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:11.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:12.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:12.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:13.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:13.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:14.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:14.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:15.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:15.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:16.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:16.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:17.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:17.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:18.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:18.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:19.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:19.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:20.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:20.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:21.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:21.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:22.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:22.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:23.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:23.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:24.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:24.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:25.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:25.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:26.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:26.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:27.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:27.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:28.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:28.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:29.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:29.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:30.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:30.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:31.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:31.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:32.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:32.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:33.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:33.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:34.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:34.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:35.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:35.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:36.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:36.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:37.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:37.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:38.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:38.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:39.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:39.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:40.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:40.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:41.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:41.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:42.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:42.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:44.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:44.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:45.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:45.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:45.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:45.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:46.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:46.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:47.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:47.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:48.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:48.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:49.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:49.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:50.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:50.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:51.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:51.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:52.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:52.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:53.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:53.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:54.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:54.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:55.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:55.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:56.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:56.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:57.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:57.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:58.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:58.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:46:59.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:46:59.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:00.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:00.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:01.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:01.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:02.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:02.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:03.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:03.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:04.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:04.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:05.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:05.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:06.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:06.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:07.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:07.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:08.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:08.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:09.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:09.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:10.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:10.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:11.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:11.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:12.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:12.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:13.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:13.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:14.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:14.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:15.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:15.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:16.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:16.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:17.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:17.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:18.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:18.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:19.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:19.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:20.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:20.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:21.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:21.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:22.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:22.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:23.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:23.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:24.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:24.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:26.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:26.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:26.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:26.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:27.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:27.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:28.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:28.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:29.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:29.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:30.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:30.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:31.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:31.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:31.886 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:47:31.886 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:47:31.887 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:47:31.887 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:31.887 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:47:31.896 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:47:31.897 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T08:47:31.897 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T08:47:31.897 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T08:47:32.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:32.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:33.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:33.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:34.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:34.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:35.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:35.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:36.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:36.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:37.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:37.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:38.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:38.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:39.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:39.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:40.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:40.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:41.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:41.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:42.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:42.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:43.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:43.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:44.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:44.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:45.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:45.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:46.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:46.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:46.898 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T08:47:46.898 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T08:47:46.898 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:47:46.899 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:46.899 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:47:47.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:47.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:47.889 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:47.920 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:47.920 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:47:47.920 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:47.920 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:47:48.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:48.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:48.942 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1614,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:47:48.975 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:48.975 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:47:49.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:49.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:50.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1614,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:47:50.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:50.036 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:47:50.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:50.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:51.021 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1614,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:47:51.040 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:47:51.040 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:51.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:47:51.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:51.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:52.172 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:47:52.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:52.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:53.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:47:53.174 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:53.174 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:47:53.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:53.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:54.190 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:54.222 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:54.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:47:54.222 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:47:54.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:54.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:47:54.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:54.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:55.238 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1614,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:47:55.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:55.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:47:55.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:55.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:56.235 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1614,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:47:56.267 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:56.267 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:47:56.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:56.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:57.297 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1614,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:47:57.318 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:47:57.318 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:47:57.318 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:47:57.319 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:47:57.319 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:47:57.319 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:47:57.337 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:47:57.337 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:47:57.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:47:57.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:47:57.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:57.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:58.408 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:58.438 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:47:58.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:58.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:47:59.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:47:59.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:00.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:00.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:01.439 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:48:01.440 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:48:01.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:01.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:02.447 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:02.478 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:02.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:02.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:03.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:03.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:04.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:04.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:05.478 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:48:05.479 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:48:05.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:05.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:06.484 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:06.514 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1614,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:06.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:06.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:07.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:07.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:08.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:08.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:09.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:48:09.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:48:09.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:09.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:10.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:10.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:11.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:11.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:12.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:12.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:12.810 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1618,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:12.833 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1618,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:48:12.833 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:48:12.833 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:48:12.833 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:48:13.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:13.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:13.839 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1618,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:48:13.860 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:48:13.860 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:48:14.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:14.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:14.868 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1618,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:48:14.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:48:15.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:15.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:16.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:16.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:17.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:17.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:18.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:18.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:19.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:19.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:20.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:20.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:21.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:21.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:22.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:22.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:23.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:23.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:24.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:24.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:25.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:25.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:26.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:26.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:27.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:27.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:28.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:28.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:29.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:29.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:30.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:30.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:31.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:31.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:32.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:32.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:33.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:33.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:34.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:34.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:35.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:35.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:36.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:36.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:37.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:37.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:38.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:38.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:39.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:39.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:40.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:40.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:41.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:41.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:42.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:42.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:43.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:43.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:44.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:44.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:45.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:45.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:46.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:46.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:47.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:47.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:48.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:48.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:49.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:49.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:50.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:50.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:51.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:51.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:52.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:52.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:53.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:53.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:54.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:54.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:55.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:55.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:56.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:56.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:57.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:57.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:58.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:58.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:48:59.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:48:59.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:00.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:00.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:01.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:01.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:02.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:02.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:03.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:03.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:04.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:04.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:05.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:05.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:06.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:06.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:07.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:07.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:08.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:08.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:09.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:09.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:10.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:10.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:11.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:11.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:12.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:12.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:13.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:13.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:14.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:14.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:15.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:15.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:16.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:16.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:17.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:17.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:18.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:18.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:19.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:19.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:20.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:20.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:21.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:21.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:22.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:22.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:23.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:23.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:24.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:24.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:25.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:25.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:26.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:26.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:27.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:27.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:28.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:28.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:29.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:29.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:30.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:30.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:31.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:31.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:32.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:32.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:33.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:33.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:34.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:34.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:35.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:35.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:36.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:36.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:37.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:37.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:38.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:38.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:39.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:39.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:40.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:40.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:41.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:41.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:42.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:42.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:44.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:43.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:44.899 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:49:44.899 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:49:44.900 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:49:44.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:44.900 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:49:44.905 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:49:44.906 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:49:44.906 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:44.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:49:44.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:44.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:45.880 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1618,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:49:45.909 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1618,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:49:45.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:45.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:46.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:46.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:47.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:47.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:48.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:48.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:49:48.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:48.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:49.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:49.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:50.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:50.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:51.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:51.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:52.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:52.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:53.751 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:49:53.774 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:49:53.774 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:49:53.774 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:53.774 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:49:54.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:54.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:54.845 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1624,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:49:54.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:54.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:49:54.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:54.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:56.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:56.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:56.096 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1624,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:49:56.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:56.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:49:57.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:57.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:57.130 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1624,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:49:57.153 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:49:57.153 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:57.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:49:58.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:58.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:58.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:49:59.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:49:59.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:49:59.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:49:59.254 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:49:59.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:50:00.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:00.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:00.279 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:00.305 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:00.305 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:50:00.305 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:50:00.306 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:00.306 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:50:01.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:01.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:01.299 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1624,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:50:01.321 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:01.322 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:50:02.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:02.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:02.333 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1624,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:50:02.365 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:02.365 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:50:03.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:03.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:03.383 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1624,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:50:03.412 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:50:03.412 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:50:03.412 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:50:03.412 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:50:03.413 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:50:03.413 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:50:03.432 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:50:03.432 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:50:03.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:03.432 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:50:04.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:04.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:04.501 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:04.522 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:05.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:05.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:06.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:06.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:07.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:07.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:07.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:07.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:50:08.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:08.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:08.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:08.558 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:09.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:09.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:10.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:10.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:11.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:11.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:11.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:11.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:50:12.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:12.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:12.572 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:12.594 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1624,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:13.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:13.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:14.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:14.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:15.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:15.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:15.595 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:15.595 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:50:16.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:16.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:17.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:17.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:18.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:18.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:18.691 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1628,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:18.713 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1628,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:50:18.713 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:50:18.714 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:18.714 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:50:19.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:19.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:19.735 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1628,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:50:19.757 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:50:19.757 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:50:20.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:20.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:20.753 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1628,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:50:20.775 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:50:21.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:21.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:22.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:22.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:23.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:23.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:24.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:24.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:25.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:25.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:26.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:26.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:27.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:27.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:28.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:28.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:29.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:29.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:30.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:30.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:31.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:31.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:32.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:32.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:33.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:33.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:34.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:34.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:35.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:35.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:36.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:36.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:37.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:37.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:38.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:38.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:39.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:39.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:40.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:40.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:41.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:41.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:42.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:42.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:43.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:43.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:43.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:43.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:44.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:44.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:45.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:45.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:46.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:46.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:47.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:47.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:49.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:49.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:50.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:50.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:51.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:51.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:52.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:52.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:53.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:53.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:54.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:54.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:55.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:55.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:56.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:56.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:57.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:57.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:58.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:58.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:50:59.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:50:59.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:00.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:00.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:01.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:01.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:02.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:02.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:03.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:03.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:04.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:04.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:05.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:05.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:06.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:06.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:07.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:07.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:08.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:08.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:09.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:09.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:10.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:10.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:11.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:11.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:12.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:12.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:13.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:13.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:14.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:14.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:15.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:15.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:16.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:16.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:17.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:17.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:18.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:18.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:19.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:19.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:20.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:20.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:21.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:21.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:22.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:22.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:23.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:23.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:24.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:24.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:25.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:25.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:26.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:26.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:27.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:27.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:28.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:28.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:29.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:29.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:30.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:30.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:31.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:31.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:32.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:32.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:33.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:33.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:34.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:34.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:35.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:35.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:36.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:36.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:37.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:37.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:38.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:38.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:39.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:39.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:40.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:40.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:41.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:41.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:42.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:42.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:43.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:43.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:44.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:44.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:45.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:45.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:46.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:46.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:47.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:47.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:48.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:48.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:49.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:49.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:50.776 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:51:50.776 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:51:50.776 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:51:50.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:51:50.777 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:51:50.784 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:51:50.784 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:51:50.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:51:50.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:51:50.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:50.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:51.777 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1628,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:51:51.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:51.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:51.799 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1628,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:51:52.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:52.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:53.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:53.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:54.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:51:54.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:51:54.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:54.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:55.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:55.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:56.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:56.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:57.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:57.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:58.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:58.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:51:59.671 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:51:59.703 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:51:59.703 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:51:59.703 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:51:59.703 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:51:59.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:51:59.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:00.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:00.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:00.892 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1634,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:00.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:00.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:52:01.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:01.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:01.932 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1634,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:01.953 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:01.954 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:52:02.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:02.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:02.920 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1634,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:52:02.943 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:52:02.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:02.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:52:03.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:03.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:04.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:52:04.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:04.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:05.074 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:52:05.074 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:05.075 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:52:05.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:05.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:06.069 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:06.093 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:06.094 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:52:06.094 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:52:06.094 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:06.094 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:52:06.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:06.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:07.098 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1634,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:07.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:07.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:52:07.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:07.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:08.148 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1634,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:08.179 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:08.179 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:52:08.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:08.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:09.183 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1634,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T08:52:09.213 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T08:52:09.213 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:52:09.213 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T08:52:09.214 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:52:09.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T08:52:09.214 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:52:09.239 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T08:52:09.239 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:52:09.240 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:09.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:52:09.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:09.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:10.305 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:10.324 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:10.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:10.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:11.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:11.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:12.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:12.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:13.325 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:13.325 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:52:13.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:13.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:14.316 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:14.340 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:14.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:14.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:15.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:15.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:16.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:16.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:17.341 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:17.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:52:17.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:17.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:18.353 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:18.375 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1634,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:18.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:18.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:19.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:19.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:20.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:20.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:21.376 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:21.377 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:52:21.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:21.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:22.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:22.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:23.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:23.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:24.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:24.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:25.435 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:25.467 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:52:25.468 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:52:25.468 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:25.468 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:52:25.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:25.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:26.475 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1638,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:26.506 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:52:26.506 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:52:26.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:26.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:27.503 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1638,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:52:27.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:52:27.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:27.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:28.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:28.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:29.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:29.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:30.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:30.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:31.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:31.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:32.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:32.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:33.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:33.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:34.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:34.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:35.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:35.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:36.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:36.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:37.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:37.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:38.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:38.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:39.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:39.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:40.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:40.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:41.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:41.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:42.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:42.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:43.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:43.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:44.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:44.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:45.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:45.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:46.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:46.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:47.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:47.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:48.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:48.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:49.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:49.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:50.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:50.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:51.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:51.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:52.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:52.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:53.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:53.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:54.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:54.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:55.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:55.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:56.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:56.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:57.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:57.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:58.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:58.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:52:59.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:52:59.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:00.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:00.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:01.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:01.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:02.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:02.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:03.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:03.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:04.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:04.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:05.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:05.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:06.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:06.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:07.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:07.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:08.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:08.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:09.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:09.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:10.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:10.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:11.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:11.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:12.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:12.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:13.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:13.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:14.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:14.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:15.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:15.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:16.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:16.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:17.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:17.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:18.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:18.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:19.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:19.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:20.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:20.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:21.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:21.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:22.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:22.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:23.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:23.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:24.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:24.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:25.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:25.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:26.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:26.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:27.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:27.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:28.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:28.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:29.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:29.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:30.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:30.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:31.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:31.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:32.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:32.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:33.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:33.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:34.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:34.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:35.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:35.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:36.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:36.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:37.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:37.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:38.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:38.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:39.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:39.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:40.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:40.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:41.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:41.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:42.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:42.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:43.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:43.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:44.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:44.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:45.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:45.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:46.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:46.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:47.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:47.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:48.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:48.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:49.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:49.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:50.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:50.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:51.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:51.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:52.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:52.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:53.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:53.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:54.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:54.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:55.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:55.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:56.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:56.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:57.535 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T08:53:57.535 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T08:53:57.535 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T08:53:57.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:53:57.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T08:53:57.542 INFO:tasks.ceph.mon.a:Started 2019-10-01T08:53:57.543 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:53:57.543 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:53:57.543 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:53:57.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:57.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:58.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:53:58.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:53:58.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:58.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:53:59.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:53:59.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:00.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:00.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:01.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:01.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:01.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:01.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:02.535 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":99,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:02.565 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1638,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":99,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:02.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:02.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:03.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:03.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:04.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:04.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:05.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:05.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:05.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:05.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:06.575 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:06.595 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:06.596 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:54:06.596 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:06.596 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:54:06.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:06.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:07.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:07.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:07.670 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1640,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:07.693 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:07.693 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:54:08.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:08.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:08.696 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1640,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:08.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:08.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:54:09.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:09.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:09.723 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1640,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:09.744 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:54:09.744 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:09.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:54:10.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:10.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:10.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:54:11.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:11.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:11.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:54:11.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:11.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:12.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:12.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:12.847 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:12.884 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:12.884 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:54:12.884 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:54:12.884 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:12.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:54:13.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:13.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:13.828 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1640,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:13.853 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:13.853 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:54:14.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:14.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:14.844 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1640,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:14.867 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:14.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:54:15.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:15.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:15.852 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1640,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:15.874 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T08:54:15.874 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:54:15.875 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T08:54:15.875 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:54:15.875 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T08:54:15.875 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:54:15.894 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T08:54:15.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:54:15.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:15.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:16.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:16.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:16.968 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:16.991 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:17.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:17.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:18.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:18.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:19.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:19.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:19.992 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:19.992 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:20.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:20.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:20.976 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:21.006 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:21.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:21.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:22.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:22.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:23.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:23.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:24.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:24.007 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:24.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:24.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:24.997 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:25.029 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1640,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:25.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:25.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:26.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:26.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:27.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:27.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:28.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:28.030 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:54:28.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:28.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:29.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:29.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:30.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:30.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:30.870 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:30.900 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:54:30.900 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:54:30.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:30.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:54:31.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:31.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:31.893 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1644,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:31.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:54:31.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:54:32.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:32.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:32.932 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1644,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:54:32.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:54:33.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:33.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:34.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:34.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:35.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:35.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:36.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:36.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:37.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:37.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:38.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:38.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:39.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:39.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:40.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:40.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:41.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:41.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:42.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:42.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:43.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:43.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:44.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:44.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:45.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:45.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:46.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:46.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:47.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:47.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:48.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:48.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:49.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:49.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:50.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:50.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:51.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:51.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:52.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:52.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:53.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:53.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:54.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:54.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:55.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:55.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:56.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:56.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:57.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:57.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:58.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:58.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:54:59.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:54:59.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:00.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:00.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:01.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:01.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:02.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:02.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:03.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:03.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:04.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:04.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:05.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:05.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:06.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:06.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:07.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:07.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:08.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:08.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:09.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:09.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:10.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:10.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:11.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:11.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:12.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:12.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:13.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:13.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:14.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:14.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:15.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:15.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:16.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:16.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:17.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:17.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:18.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:18.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:19.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:19.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:20.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:20.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:21.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:21.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:22.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:22.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:23.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:23.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:24.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:24.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:25.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:25.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:26.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:26.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:27.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:27.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:28.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:28.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:29.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:29.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:30.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:30.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:31.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:31.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:32.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:32.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:33.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:33.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:34.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:34.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:35.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:35.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:36.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:36.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:37.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:37.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:38.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:38.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:39.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:39.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:40.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:40.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:41.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:41.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:42.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:42.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:43.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:43.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:44.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:44.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:45.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:45.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:46.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:46.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:47.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:47.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:48.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:48.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:49.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:49.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:50.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:50.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:51.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:51.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:52.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:52.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:53.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:53.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:54.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:54.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:55.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:55.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:56.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:56.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:57.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:57.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:58.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:58.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:55:59.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:55:59.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:00.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:00.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:01.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:01.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:02.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:02.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:02.955 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T08:56:02.956 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T08:56:02.956 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T08:56:02.956 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:02.956 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T08:56:02.962 INFO:tasks.ceph.mon.c:Started 2019-10-01T08:56:02.963 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:56:02.963 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:02.963 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:03.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:03.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:03.969 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:03.997 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:04.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:04.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:05.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:05.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:06.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:06.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:06.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:06.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:07.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:07.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:07.999 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:08.022 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1644,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:08.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:08.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:09.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:09.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:10.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:10.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:11.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:11.023 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:11.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:11.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:12.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:12.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:13.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:13.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:14.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:14.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:15.076 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:15.102 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:15.102 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:56:15.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:15.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:56:15.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:15.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:16.096 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1650,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:16.119 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:16.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:56:16.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:16.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:17.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:17.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:18.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:18.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:19.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:19.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:20.061 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1650,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:20.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:20.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:56:20.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:20.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:21.090 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1650,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:21.112 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:56:21.112 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:21.113 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:56:21.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:21.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:22.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:56:22.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:22.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:23.344 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:56:23.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:23.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:23.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:23.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:24.304 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:24.326 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:24.326 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:56:24.326 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:56:24.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:24.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:56:24.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:24.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:25.307 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1650,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:25.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:25.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:56:25.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:25.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:26.332 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1650,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:26.356 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:26.356 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:56:26.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:26.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:27.338 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1650,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:27.368 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:56:27.368 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:56:27.368 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:56:27.368 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:56:27.368 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:56:27.369 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:56:27.408 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:56:27.408 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:56:27.409 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:27.409 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:27.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:27.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:28.426 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:28.449 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:28.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:28.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:29.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:29.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:30.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:30.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:31.449 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:31.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:31.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:31.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:32.431 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:32.453 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1650,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:32.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:32.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:33.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:33.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:34.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:34.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:35.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:35.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:56:35.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:35.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:36.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:36.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:37.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:37.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:38.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:38.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:39.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:39.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:40.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:40.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:41.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:41.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:42.966 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:42.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:42.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:42.997 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:56:42.997 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:56:42.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:42.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:56:43.997 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1652,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:44.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:44.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:44.021 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:56:44.021 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:56:44.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:44.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:45.005 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1652,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:56:45.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:56:46.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:46.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:47.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:47.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:48.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:48.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:49.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:49.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:50.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:50.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:51.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:51.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:51.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:51.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:52.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:52.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:53.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:53.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:54.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:54.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:55.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:55.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:56.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:56.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:57.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:57.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:58.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:58.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:56:59.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:56:59.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:00.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:00.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:01.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:01.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:02.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:02.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:03.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:03.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:04.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:04.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:05.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:05.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:06.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:06.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:07.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:07.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:08.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:08.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:09.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:09.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:10.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:10.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:11.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:11.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:12.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:12.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:13.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:13.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:14.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:14.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:15.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:15.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:16.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:16.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:17.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:17.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:18.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:18.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:19.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:19.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:20.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:20.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:21.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:21.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:22.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:22.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:23.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:23.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:24.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:24.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:25.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:25.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:26.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:26.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:27.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:27.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:28.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:28.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:29.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:29.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:30.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:30.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:31.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:31.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:32.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:32.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:33.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:33.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:34.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:34.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:35.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:35.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:36.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:36.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:37.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:37.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:38.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:38.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:39.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:39.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:40.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:40.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:41.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:41.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:42.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:42.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:43.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:43.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:44.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:44.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:45.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:45.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:46.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:46.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:47.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:47.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:48.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:48.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:49.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:49.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:50.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:50.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:51.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:51.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:52.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:52.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:53.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:53.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:54.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:54.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:55.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:55.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:56.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:56.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:57.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:57.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:58.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:58.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:57:59.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:57:59.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:00.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:00.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:01.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:01.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:02.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:02.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:03.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:03.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:04.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:04.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:05.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:05.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:06.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:06.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:07.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:07.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:08.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:08.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:09.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:09.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:10.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:10.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:11.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:11.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:12.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:12.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:13.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:13.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:14.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:14.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:15.029 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:58:15.030 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T08:58:15.030 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T08:58:15.030 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T08:58:15.030 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T08:58:15.036 INFO:tasks.ceph.mon.b:Started 2019-10-01T08:58:15.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:58:15.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:15.037 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:15.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:15.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:16.028 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:16.050 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:16.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:16.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:17.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:17.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:18.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:18.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:19.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:19.051 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:19.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:19.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:20.045 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:20.067 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:20.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:20.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:21.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:21.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:22.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:22.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:23.068 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:23.069 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:23.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:23.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:24.047 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:24.070 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:24.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:24.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:25.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:25.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:26.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:26.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:27.070 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:27.071 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:27.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:27.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:28.101 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:28.123 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1652,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:28.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:28.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:29.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:29.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:30.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:30.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:31.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:31.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:31.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:31.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:32.124 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:32.145 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:32.146 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:58:32.146 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:32.146 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:58:32.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:32.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:33.102 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1656,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:33.123 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:33.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:58:33.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:33.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:34.054 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1656,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:34.085 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:34.085 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:58:34.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:34.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:35.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1656,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:35.102 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T08:58:35.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:35.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T08:58:35.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:35.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:36.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T08:58:36.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:36.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:37.238 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T08:58:37.238 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:37.238 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:37.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:37.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:38.228 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:38.258 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:38.258 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T08:58:38.258 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T08:58:38.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:38.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:58:38.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:38.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:39.217 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1656,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:39.233 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:39.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:58:39.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:39.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:40.237 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1656,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:40.267 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:40.268 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T08:58:40.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:40.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:41.252 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1656,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:41.282 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T08:58:41.282 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T08:58:41.282 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T08:58:41.282 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T08:58:41.283 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T08:58:41.283 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T08:58:41.301 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T08:58:41.302 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T08:58:41.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:41.302 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:41.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:41.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:42.306 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:42.328 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:42.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:42.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:43.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:43.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:44.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:44.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:45.330 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:45.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:45.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:45.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:46.300 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:46.321 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:46.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:46.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:47.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:47.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:48.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:48.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:49.322 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:49.322 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:49.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:49.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:50.325 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:50.347 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1656,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:51.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:51.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:52.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:51.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:53.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:53.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:53.347 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:53.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T08:58:54.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:54.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:55.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:55.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:56.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:56.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:56.425 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1658,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:56.453 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1658,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T08:58:56.453 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T08:58:56.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:56.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T08:58:57.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:57.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:57.442 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1658,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:57.460 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T08:58:57.460 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T08:58:58.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:58.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:58:58.481 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1658,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T08:58:58.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T08:58:59.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:58:59.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:00.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:00.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:01.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:01.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:02.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:02.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:03.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:03.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:04.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:04.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:05.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:05.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:06.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:06.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:07.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:07.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:08.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:08.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:09.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:09.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:09.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:09.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:10.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:10.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:11.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:11.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:12.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:12.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:13.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:13.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:15.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:15.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:16.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:16.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:17.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:17.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:18.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:18.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:19.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:19.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:20.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:20.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:21.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:21.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:22.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:22.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:23.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:23.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:24.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:24.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:25.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:25.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:26.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:26.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:27.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:27.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:28.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:28.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:29.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:29.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:30.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:30.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:31.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:31.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:32.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:32.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:33.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:33.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:34.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:34.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:35.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:35.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:36.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:36.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:37.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:37.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:38.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:38.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:39.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:39.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:40.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:40.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:41.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:41.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:42.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:42.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:43.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:43.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:44.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:44.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:45.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:45.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:46.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:46.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:47.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:47.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:48.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:48.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:49.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:49.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:50.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:50.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:51.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:51.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:52.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:52.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:53.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:53.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:54.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:54.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:55.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:55.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:56.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:56.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:57.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:57.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:58.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:58.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T08:59:59.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T08:59:59.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:00.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:00.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:01.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:01.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:02.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:02.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:03.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:03.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:04.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:04.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:05.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:05.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:06.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:06.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:07.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:07.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:08.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:08.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:09.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:09.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:10.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:10.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:11.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:11.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:12.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:12.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:13.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:13.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:14.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:14.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:15.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:15.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:16.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:16.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:17.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:17.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:18.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:18.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:19.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:19.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:20.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:20.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:21.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:21.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:22.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:22.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:23.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:23.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:24.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:24.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:25.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:25.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:26.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:26.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:27.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:27.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:28.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:28.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:28.512 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:00:28.513 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:00:28.513 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:00:28.513 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:00:28.513 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:00:28.523 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:00:28.524 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:00:28.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:28.524 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:00:29.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:29.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:29.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1658,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:29.553 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1658,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:30.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:30.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:31.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:31.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:32.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:32.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:32.555 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:32.555 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:00:33.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:33.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:34.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:34.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:35.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:35.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:36.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:36.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:37.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:37.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:38.213 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1662,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:38.235 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1662,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:38.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:38.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:39.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:39.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:40.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:40.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:41.235 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:41.235 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:00:41.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:41.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:42.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:42.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:43.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:43.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:44.224 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1666,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:44.246 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1666,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:44.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:44.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:45.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:45.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:46.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:46.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:47.248 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:47.249 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:00:47.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:47.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:48.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:48.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:49.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:49.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:50.242 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1670,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:50.265 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1670,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:50.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:50.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:51.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:51.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:52.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:52.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:53.272 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:53.273 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:00:53.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:53.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:54.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:54.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:54.911 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:54.940 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:00:54.940 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:00:54.941 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:54.941 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:00:55.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:55.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:55.938 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1674,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:00:55.971 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:55.971 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:00:56.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:56.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:56.963 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1674,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:00:56.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:56.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:00:57.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:57.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:57.993 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1674,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:00:58.022 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:00:58.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:00:58.023 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:00:58.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:58.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:00:59.136 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:00:59.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:00:59.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:00.137 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:01:00.137 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:00.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:01:00.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:00.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:01.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:01.157 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:01.158 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:01:01.158 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:01:01.158 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:01.158 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:01:01.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:01.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:02.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1674,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:01:02.145 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:02.145 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:01:02.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:02.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:03.215 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1674,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:01:03.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:03.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:03.238 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:03.238 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:01:04.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:04.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:04.313 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1674,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:01:04.335 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:01:04.335 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:01:04.335 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:01:04.335 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:01:04.335 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:01:04.335 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:01:04.354 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:01:04.354 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:01:04.354 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:04.354 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:01:05.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:05.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:05.442 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:05.469 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:06.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:06.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:07.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:07.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:08.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:08.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:08.470 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:08.470 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:01:09.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:09.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:09.448 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:09.466 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:10.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:10.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:11.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:11.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:12.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:12.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:12.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:12.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:01:13.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:13.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:13.456 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:13.478 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1674,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:14.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:14.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:15.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:15.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:16.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:16.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:16.478 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:16.479 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:01:17.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:17.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:18.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:18.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:19.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:19.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:20.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:20.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:20.499 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1678,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:20.521 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1678,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:01:20.521 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:01:20.522 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:20.522 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:01:21.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:21.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:21.515 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1678,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:01:21.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:01:21.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:01:22.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:22.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:22.562 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1678,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:01:22.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:01:23.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:23.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:24.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:24.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:25.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:25.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:26.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:26.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:27.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:27.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:28.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:28.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:29.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:29.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:30.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:30.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:31.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:31.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:32.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:32.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:33.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:33.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:34.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:34.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:35.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:35.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:36.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:36.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:37.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:37.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:38.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:38.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:39.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:39.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:40.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:40.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:41.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:41.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:42.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:42.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:43.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:43.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:44.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:44.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:45.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:45.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:46.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:46.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:47.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:47.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:48.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:48.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:49.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:49.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:50.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:50.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:51.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:51.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:52.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:52.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:53.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:53.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:54.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:54.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:55.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:55.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:56.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:56.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:57.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:57.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:58.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:58.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:01:59.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:01:59.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:00.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:00.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:01.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:01.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:02.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:02.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:03.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:03.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:04.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:04.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:05.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:05.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:06.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:06.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:07.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:07.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:08.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:08.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:09.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:09.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:10.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:10.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:11.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:11.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:12.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:12.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:13.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:13.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:14.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:14.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:15.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:15.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:16.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:16.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:17.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:17.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:18.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:18.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:19.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:19.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:20.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:20.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:21.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:21.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:22.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:22.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:23.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:23.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:24.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:24.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:25.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:25.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:26.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:26.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:27.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:27.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:28.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:28.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:29.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:29.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:30.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:30.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:31.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:31.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:32.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:32.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:33.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:33.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:34.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:34.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:35.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:35.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:36.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:36.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:37.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:37.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:38.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:38.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:39.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:39.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:40.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:40.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:41.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:41.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:42.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:42.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:43.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:43.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:44.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:44.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:45.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:45.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:46.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:46.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:47.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:47.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:48.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:48.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:49.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:49.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:50.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:50.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:51.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:51.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:52.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:52.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:52.593 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:02:52.594 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:02:52.594 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:02:52.594 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:02:52.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:02:52.600 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:02:52.600 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:02:52.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:02:52.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:02:53.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:53.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:53.580 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1678,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:02:53.602 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1678,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:02:54.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:54.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:55.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:55.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:56.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:56.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:56.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:02:56.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:02:57.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:57.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:58.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:58.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:02:59.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:02:59.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:00.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:00.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:01.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:01.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:02.150 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:02.173 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:02.173 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:03:02.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:02.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:03:02.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:02.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:03.167 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1686,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:03.197 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:03.197 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:03:03.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:03.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:04.254 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1686,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:04.275 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:04.275 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:03:04.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:04.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:05.267 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1686,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:05.296 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:03:05.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:05.297 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:03:05.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:05.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:06.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:06.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:06.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:03:07.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:07.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:07.482 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:03:07.482 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:07.483 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:03:08.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:08.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:08.504 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:08.526 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:08.526 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:03:08.526 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:03:08.527 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:08.527 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:03:09.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:09.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:09.518 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1686,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:09.547 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:09.547 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:03:10.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:10.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:10.544 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1686,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:10.568 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:10.569 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:03:11.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:11.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:11.546 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1686,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:11.565 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:03:11.565 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:03:11.565 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:03:11.566 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:03:11.566 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:03:11.566 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:03:11.585 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:03:11.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:03:11.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:11.586 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:03:12.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:12.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:12.624 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:12.646 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:13.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:13.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:14.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:14.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:15.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:15.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:15.646 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:15.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:03:16.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:16.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:16.650 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:16.675 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:17.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:17.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:18.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:18.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:19.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:19.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:19.676 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:19.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:03:20.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:20.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:20.663 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:20.684 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1686,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:21.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:21.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:22.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:22.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:23.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:23.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:23.685 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:23.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:03:24.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:24.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:25.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:25.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:26.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:26.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:26.669 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1690,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:26.692 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1690,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:03:26.692 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:03:26.692 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:26.692 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:03:27.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:27.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:27.690 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1690,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:27.712 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:03:27.712 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:03:28.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:28.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:28.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1690,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:03:28.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:03:29.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:29.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:30.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:30.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:31.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:31.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:32.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:32.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:33.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:33.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:34.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:34.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:35.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:35.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:36.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:36.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:37.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:37.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:38.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:38.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:39.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:39.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:40.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:40.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:41.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:41.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:42.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:42.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:43.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:43.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:44.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:44.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:45.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:45.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:46.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:46.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:47.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:47.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:48.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:48.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:49.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:49.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:50.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:50.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:51.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:51.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:52.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:52.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:53.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:53.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:54.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:54.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:55.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:55.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:56.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:56.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:57.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:57.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:58.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:58.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:03:59.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:03:59.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:00.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:00.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:01.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:01.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:02.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:02.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:03.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:03.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:04.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:04.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:05.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:05.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:06.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:06.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:07.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:07.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:08.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:08.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:09.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:09.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:10.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:10.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:11.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:11.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:12.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:12.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:13.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:13.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:14.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:14.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:15.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:15.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:16.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:16.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:17.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:17.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:18.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:18.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:19.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:19.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:20.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:20.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:21.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:21.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:22.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:22.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:23.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:23.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:24.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:24.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:25.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:25.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:26.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:26.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:27.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:27.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:28.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:28.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:29.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:29.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:30.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:30.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:31.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:31.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:32.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:32.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:33.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:33.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:34.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:34.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:35.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:35.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:36.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:36.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:37.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:37.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:38.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:38.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:39.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:39.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:40.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:40.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:41.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:41.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:42.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:42.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:43.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:43.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:44.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:44.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:45.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:45.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:46.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:46.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:47.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:47.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:48.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:48.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:49.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:49.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:50.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:50.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:51.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:51.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:52.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:52.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:53.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:53.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:54.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:54.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:55.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:55.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:56.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:56.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:57.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:57.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:58.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:58.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:58.752 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:04:58.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:04:58.752 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:04:58.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:04:58.753 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:04:58.758 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:04:58.758 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:04:58.758 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:04:58.759 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:04:59.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:04:59.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:04:59.751 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1690,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:04:59.773 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1690,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:00.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:00.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:01.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:01.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:02.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:02.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:02.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:02.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:05:03.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:03.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:04.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:04.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:05.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:05.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:06.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:06.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:07.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:07.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:07.686 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:07.707 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:07.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:05:07.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:07.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:05:08.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:08.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:08.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1696,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:08.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:08.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:05:09.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:09.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:09.750 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1696,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:09.780 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:09.780 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:05:10.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:10.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:10.701 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1696,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:10.724 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:05:10.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:10.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:05:11.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:11.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:11.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:05:12.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:12.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:12.824 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:05:12.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:12.825 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:05:13.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:13.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:13.798 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:13.820 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1696,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:13.820 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:05:13.820 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:05:13.821 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:13.821 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:05:14.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:14.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:14.803 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1696,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:14.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:14.826 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:05:15.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:15.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:15.762 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1696,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:15.792 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:15.792 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:05:16.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:16.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:16.801 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1696,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:16.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:05:16.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T09:05:16.831 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:05:16.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:05:16.832 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:05:16.832 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:05:16.851 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:05:16.851 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:05:16.852 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:05:16.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:05:17.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:17.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:18.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:18.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:19.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:19.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:20.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:20.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:21.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:21.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:22.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:22.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:23.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:23.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:24.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:24.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:25.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:25.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:26.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:26.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:27.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:27.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:28.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:28.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:29.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:29.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:30.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:30.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:31.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:31.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:31.853 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:05:31.853 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:05:31.853 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:05:31.853 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:31.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:05:32.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:32.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:33.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:33.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:34.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:34.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:35.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:35.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:36.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:36.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:37.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:37.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:38.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:38.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:39.082 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:39.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1700,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:05:39.115 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:05:39.115 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:39.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:05:39.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:39.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:40.125 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1700,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:40.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:05:40.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:05:40.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:40.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:41.146 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1700,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:05:41.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:05:41.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:41.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:42.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:42.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:43.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:43.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:44.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:44.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:45.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:45.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:46.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:46.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:47.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:47.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:48.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:48.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:49.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:49.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:50.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:50.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:51.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:51.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:52.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:52.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:53.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:53.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:54.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:54.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:55.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:55.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:56.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:56.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:57.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:57.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:58.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:58.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:05:59.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:05:59.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:00.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:00.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:01.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:01.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:02.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:02.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:03.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:03.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:04.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:04.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:05.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:05.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:06.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:06.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:07.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:07.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:08.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:08.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:09.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:09.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:10.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:10.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:11.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:11.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:12.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:12.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:13.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:13.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:14.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:14.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:15.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:15.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:16.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:16.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:17.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:17.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:18.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:18.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:19.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:19.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:20.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:20.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:21.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:21.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:22.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:22.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:23.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:23.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:24.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:24.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:25.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:25.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:26.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:26.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:27.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:27.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:28.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:28.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:29.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:29.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:30.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:30.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:31.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:31.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:32.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:32.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:33.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:33.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:34.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:34.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:35.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:35.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:36.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:36.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:37.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:37.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:38.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:38.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:39.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:39.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:40.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:40.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:41.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:41.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:42.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:42.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:43.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:43.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:44.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:44.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:45.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:45.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:46.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:46.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:47.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:47.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:48.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:48.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:49.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:49.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:50.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:50.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:51.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:51.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:52.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:52.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:53.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:53.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:54.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:54.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:55.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:55.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:56.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:56.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:57.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:57.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:58.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:58.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:06:59.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:06:59.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:00.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:00.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:01.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:01.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:02.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:02.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:03.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:03.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:04.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:04.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:05.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:05.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:06.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:06.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:07.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:07.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:08.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:08.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:09.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:09.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:10.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:10.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:11.170 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:07:11.170 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:07:11.170 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:07:11.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:11.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:07:11.177 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:07:11.178 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:07:11.178 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:07:11.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:07:11.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:11.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:12.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:12.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:13.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:13.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:14.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:14.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:15.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:15.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:16.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:16.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:17.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:17.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:18.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:18.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:19.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:19.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:20.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:20.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:21.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:21.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:22.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:22.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:23.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:23.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:24.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:24.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:25.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:25.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:26.179 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:07:26.180 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:07:26.180 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:07:26.180 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:26.180 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:26.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:26.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:27.192 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:27.220 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:27.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:07:27.221 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:27.221 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:07:27.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:27.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:28.215 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1704,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:28.239 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:28.239 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:07:28.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:28.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:29.250 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1704,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:29.280 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:29.280 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:07:29.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:29.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:30.294 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1704,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:30.317 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:07:30.317 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:30.317 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:07:30.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:30.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:31.439 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:07:31.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:31.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:32.440 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:07:32.440 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:32.440 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:32.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:32.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:33.422 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:33.453 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:33.453 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:07:33.454 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:07:33.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:33.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:07:33.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:33.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:34.462 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1704,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:34.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:34.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:07:34.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:34.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:35.526 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1704,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:35.556 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:35.557 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:07:35.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:35.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:36.575 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1704,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:36.598 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:07:36.598 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:07:36.598 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:07:36.598 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:07:36.598 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:07:36.599 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:07:36.617 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:07:36.617 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:07:36.617 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:36.618 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:36.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:36.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:37.671 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:37.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:37.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:37.693 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:38.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:38.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:39.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:39.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:40.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:40.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:40.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:40.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:41.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:41.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:41.694 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:41.716 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:42.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:42.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:43.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:43.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:44.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:44.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:44.718 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:44.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:45.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:45.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:45.728 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:45.759 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1704,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:46.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:46.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:47.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:47.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:48.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:48.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:48.760 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:48.760 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:07:49.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:49.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:50.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:50.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:51.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:51.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:52.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:52.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:52.807 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1708,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:52.828 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1708,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:07:52.829 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:07:52.829 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:52.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:07:53.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:53.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:53.788 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1708,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:53.810 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:07:53.810 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:07:54.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:54.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:54.795 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1708,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:07:54.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:07:55.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:55.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:56.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:56.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:57.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:57.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:58.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:58.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:07:59.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:07:59.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:00.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:00.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:01.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:01.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:02.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:02.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:03.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:03.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:04.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:04.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:05.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:05.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:06.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:06.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:07.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:07.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:08.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:08.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:09.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:09.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:10.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:10.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:11.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:11.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:12.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:12.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:13.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:13.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:14.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:14.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:15.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:15.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:16.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:16.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:17.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:17.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:18.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:18.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:19.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:19.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:20.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:20.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:21.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:21.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:22.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:22.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:23.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:23.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:24.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:24.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:25.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:25.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:26.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:26.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:27.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:27.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:28.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:28.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:29.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:29.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:30.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:30.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:31.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:31.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:32.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:32.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:33.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:33.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:34.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:34.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:35.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:35.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:36.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:36.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:37.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:37.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:38.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:38.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:39.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:39.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:40.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:40.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:41.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:41.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:42.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:42.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:43.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:43.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:44.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:44.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:45.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:45.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:46.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:46.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:47.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:47.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:48.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:48.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:49.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:49.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:50.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:50.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:51.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:51.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:52.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:52.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:53.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:53.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:54.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:54.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:55.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:55.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:56.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:56.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:57.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:57.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:58.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:58.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:08:59.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:08:59.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:00.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:00.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:01.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:01.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:02.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:02.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:03.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:03.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:04.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:04.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:05.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:05.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:06.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:06.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:07.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:07.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:08.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:08.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:09.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:09.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:10.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:10.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:11.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:11.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:12.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:12.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:13.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:13.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:14.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:14.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:15.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:15.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:16.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:16.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:17.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:17.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:18.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:18.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:19.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:19.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:20.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:20.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:21.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:21.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:22.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:22.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:23.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:23.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:24.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:24.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:24.818 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:09:24.818 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:09:24.818 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:09:24.819 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:24.819 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:09:24.825 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:09:24.825 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:09:24.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:24.825 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:09:25.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:25.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:25.786 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1708,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:25.804 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1708,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:26.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:26.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:27.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:27.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:28.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:28.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:28.810 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:28.810 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:09:29.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:29.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:29.800 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1714,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:29.832 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1714,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:29.832 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:09:29.833 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:29.833 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:09:30.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:30.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:30.820 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1714,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:30.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:30.846 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:09:31.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:31.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:31.844 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1714,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:31.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:31.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:09:32.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:32.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:32.871 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1714,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:32.892 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:09:32.892 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:32.892 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:09:33.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:33.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:33.978 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:09:34.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:34.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:34.979 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:09:34.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:34.980 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:09:35.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:35.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:35.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1714,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:36.013 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1714,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:09:36.013 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:09:36.013 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:09:36.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:36.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:09:36.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:36.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:37.009 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1714,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:37.037 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:37.037 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:09:37.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:37.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:38.017 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1714,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:38.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:38.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:09:38.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:38.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:39.046 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1714,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:09:39.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:09:39.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T09:09:39.068 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:09:39.068 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:09:39.069 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:09:39.069 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:09:39.099 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:09:39.099 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:09:39.099 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:09:39.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:09:39.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:39.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:40.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:40.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:41.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:41.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:42.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:42.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:43.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:43.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:44.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:44.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:45.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:45.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:46.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:46.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:47.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:47.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:48.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:48.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:49.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:49.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:50.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:50.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:51.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:51.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:52.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:52.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:53.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:53.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:54.100 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:09:54.101 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:09:54.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:09:54.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:09:54.101 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:09:54.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:54.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:55.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:55.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:56.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:56.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:57.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:57.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:58.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:58.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:09:59.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:09:59.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:00.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:00.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:01.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:01.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:01.292 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1716,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:10:01.316 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1716,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:10:01.317 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:10:01.317 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:10:01.317 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:10:02.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:02.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:02.347 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1716,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:10:02.372 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:10:02.372 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:10:03.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:03.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:03.376 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1716,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:10:03.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:10:04.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:04.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:05.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:05.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:06.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:06.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:07.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:07.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:08.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:08.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:09.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:09.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:10.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:10.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:11.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:11.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:12.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:12.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:13.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:13.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:14.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:14.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:15.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:15.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:16.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:16.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:17.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:17.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:18.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:18.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:19.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:19.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:20.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:20.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:21.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:21.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:22.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:22.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:23.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:23.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:24.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:24.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:25.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:25.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:26.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:26.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:27.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:27.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:28.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:28.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:29.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:29.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:30.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:30.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:31.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:31.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:32.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:32.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:33.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:33.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:34.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:34.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:35.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:35.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:36.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:36.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:37.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:37.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:38.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:38.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:39.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:39.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:40.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:40.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:41.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:41.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:42.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:42.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:43.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:43.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:44.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:44.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:45.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:45.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:46.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:46.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:47.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:47.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:48.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:48.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:49.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:49.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:50.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:50.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:51.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:51.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:52.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:52.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:53.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:53.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:54.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:54.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:55.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:55.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:56.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:56.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:57.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:57.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:58.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:58.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:10:59.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:10:59.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:00.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:00.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:01.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:01.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:02.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:02.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:03.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:03.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:04.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:04.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:05.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:05.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:06.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:06.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:07.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:07.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:08.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:08.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:09.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:09.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:10.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:10.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:11.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:11.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:12.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:12.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:13.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:13.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:14.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:14.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:15.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:15.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:16.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:16.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:17.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:17.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:18.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:18.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:19.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:19.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:20.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:20.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:21.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:21.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:22.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:22.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:23.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:23.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:24.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:24.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:25.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:25.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:26.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:26.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:27.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:27.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:28.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:28.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:29.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:29.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:30.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:30.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:31.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:31.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:32.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:32.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:33.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:33.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:33.408 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:11:33.408 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:11:33.409 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:11:33.409 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:11:33.409 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:11:33.417 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:11:33.417 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:11:33.417 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:11:33.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:11:34.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:34.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:35.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:35.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:36.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:36.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:37.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:37.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:38.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:38.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:39.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:39.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:40.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:40.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:41.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:41.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:42.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:42.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:43.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:43.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:44.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:44.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:45.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:45.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:46.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:46.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:47.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:47.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:48.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:48.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:48.417 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:11:48.418 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:11:48.418 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:11:48.418 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:48.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:11:49.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:49.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:49.470 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1724,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:11:49.492 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1724,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:11:49.492 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:11:49.493 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:49.493 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:11:50.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:50.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:50.479 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1724,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:11:50.501 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:50.501 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:11:51.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:51.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:51.527 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1724,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:11:51.559 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:51.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:11:52.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:52.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:52.578 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1724,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:11:52.600 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:11:52.601 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:52.601 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:11:53.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:53.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:53.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:11:53.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:53.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:54.684 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:11:54.685 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:54.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:11:55.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:55.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:55.678 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1724,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:11:55.700 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1724,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:11:55.700 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:11:55.700 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:11:55.701 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:55.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:11:56.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:56.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:56.683 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1724,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:11:56.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:56.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:11:57.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:57.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:57.703 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1724,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:11:57.734 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:11:57.734 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:11:58.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:58.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:11:58.685 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1724,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:11:58.715 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:11:58.715 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T09:11:58.715 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:11:58.715 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:11:58.715 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:11:58.716 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:11:58.744 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:11:58.744 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:11:58.745 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:11:58.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:11:59.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:11:59.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:00.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:00.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:01.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:01.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:02.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:02.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:03.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:03.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:04.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:04.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:05.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:05.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:06.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:06.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:07.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:07.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:08.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:08.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:09.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:09.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:10.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:10.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:11.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:11.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:12.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:12.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:13.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:12.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:13.746 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:12:13.746 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:12:13.746 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:12:13.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:12:13.747 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:12:13.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:13.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:15.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:15.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:16.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:16.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:17.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:17.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:18.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:18.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:19.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:19.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:19.408 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1726,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:12:19.430 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1726,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:12:19.430 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:12:19.430 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:12:19.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:12:20.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:20.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:20.424 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1726,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:12:20.446 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:12:20.446 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:12:21.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:21.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:21.439 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1726,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:12:21.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:12:22.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:22.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:23.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:23.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:24.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:24.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:25.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:25.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:26.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:26.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:27.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:27.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:28.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:28.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:29.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:29.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:30.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:30.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:31.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:31.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:32.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:32.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:33.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:33.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:34.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:34.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:35.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:35.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:36.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:36.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:37.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:37.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:38.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:38.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:39.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:39.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:40.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:40.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:41.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:41.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:42.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:42.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:43.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:43.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:44.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:44.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:45.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:45.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:46.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:46.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:47.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:47.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:48.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:48.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:49.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:49.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:50.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:50.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:51.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:51.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:52.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:52.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:53.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:53.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:54.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:54.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:55.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:55.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:56.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:56.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:57.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:57.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:58.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:58.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:12:59.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:12:59.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:00.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:00.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:01.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:01.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:02.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:02.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:03.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:03.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:04.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:04.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:05.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:05.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:06.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:06.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:07.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:07.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:08.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:08.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:09.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:09.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:10.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:10.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:11.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:11.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:12.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:12.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:13.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:13.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:14.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:14.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:15.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:15.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:16.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:16.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:17.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:17.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:18.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:18.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:19.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:19.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:20.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:20.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:21.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:21.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:22.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:22.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:23.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:23.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:24.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:24.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:25.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:25.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:26.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:26.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:27.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:27.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:28.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:28.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:29.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:29.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:30.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:30.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:31.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:31.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:32.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:32.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:33.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:33.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:34.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:34.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:35.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:35.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:36.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:36.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:37.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:37.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:38.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:38.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:39.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:39.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:40.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:40.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:41.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:41.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:42.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:42.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:43.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:43.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:44.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:44.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:45.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:45.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:46.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:46.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:47.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:47.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:48.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:48.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:49.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:49.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:50.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:50.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:51.464 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:13:51.464 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:13:51.464 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:13:51.465 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:13:51.465 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:13:51.477 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:13:51.477 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:13:51.477 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:13:51.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:13:51.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:51.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:52.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:52.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:53.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:53.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:54.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:54.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:55.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:55.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:56.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:56.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:57.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:57.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:58.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:58.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:13:59.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:13:59.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:00.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:00.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:01.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:01.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:02.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:02.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:03.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:03.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:04.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:04.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:05.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:05.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:06.483 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:14:06.484 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:14:06.484 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:14:06.484 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:06.484 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:06.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:06.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:07.500 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:07.523 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:07.523 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:14:07.523 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:07.523 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:14:07.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:07.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:08.508 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1736,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:14:08.529 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:08.529 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:14:08.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:08.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:09.538 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1736,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:14:09.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:09.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:14:09.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:09.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:10.559 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1736,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:14:10.582 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:14:10.582 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:10.582 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:14:10.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:10.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:11.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:14:11.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:11.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:12.665 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:14:12.666 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:12.666 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:12.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:12.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:13.685 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:13.717 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:13.717 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:14:13.717 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:14:13.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:13.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:14:13.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:13.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:14.713 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1736,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:14:14.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:14.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:14:14.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:14.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:15.764 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1736,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:14:15.795 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:15.795 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:14:15.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:15.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:16.795 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1736,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:14:16.822 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:14:16.822 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:14:16.822 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:14:16.823 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:14:16.823 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:14:16.823 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:14:16.841 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:14:16.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:14:16.841 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:16.841 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:16.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:16.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:17.824 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:17.846 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:17.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:17.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:18.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:18.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:19.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:19.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:20.847 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:20.847 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:20.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:20.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:21.860 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:21.878 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:21.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:21.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:23.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:23.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:24.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:24.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:24.878 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:24.878 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:25.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:25.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:25.872 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:25.900 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1736,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:26.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:26.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:27.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:27.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:28.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:28.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:28.901 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:28.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:14:29.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:29.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:30.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:30.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:31.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:31.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:32.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:32.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:32.455 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1738,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:32.477 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1738,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:14:32.477 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:14:32.477 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:32.478 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:14:32.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:32.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:33.432 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1738,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:14:33.455 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:14:33.455 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:14:33.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:33.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:34.453 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1738,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:14:34.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:14:34.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:34.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:35.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:35.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:36.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:36.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:37.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:37.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:38.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:38.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:39.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:39.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:40.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:40.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:41.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:41.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:42.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:42.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:43.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:43.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:44.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:44.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:45.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:45.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:46.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:46.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:47.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:47.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:48.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:48.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:49.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:49.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:50.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:50.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:51.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:51.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:52.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:52.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:53.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:53.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:54.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:54.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:55.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:55.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:56.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:56.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:57.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:57.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:58.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:58.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:14:59.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:14:59.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:00.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:00.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:01.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:01.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:02.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:02.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:03.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:03.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:04.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:04.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:05.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:05.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:06.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:06.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:07.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:07.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:08.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:08.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:09.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:09.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:10.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:10.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:11.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:11.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:12.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:12.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:13.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:13.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:14.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:14.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:15.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:15.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:16.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:16.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:17.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:17.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:18.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:18.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:19.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:19.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:20.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:20.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:21.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:21.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:22.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:22.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:23.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:23.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:24.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:24.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:25.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:25.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:26.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:26.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:27.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:27.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:28.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:28.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:29.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:29.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:30.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:30.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:31.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:31.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:32.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:32.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:33.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:33.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:34.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:34.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:35.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:35.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:36.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:36.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:37.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:37.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:38.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:38.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:39.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:39.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:40.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:40.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:41.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:41.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:42.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:42.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:43.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:43.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:44.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:44.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:45.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:45.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:46.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:46.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:47.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:47.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:48.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:48.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:49.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:49.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:50.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:50.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:51.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:51.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:52.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:52.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:53.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:53.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:54.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:54.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:55.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:55.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:56.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:56.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:57.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:57.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:58.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:58.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:15:59.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:15:59.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:00.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:00.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:01.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:01.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:02.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:02.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:03.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:03.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:04.485 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:16:04.485 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:16:04.485 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:16:04.486 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:16:04.486 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:16:04.491 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:16:04.492 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:16:04.492 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:04.492 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:04.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:04.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:05.478 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1738,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:05.508 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1738,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:05.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:05.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:06.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:06.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:07.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:07.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:08.509 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:08.509 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:08.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:08.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:09.510 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:09.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:09.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:09.532 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:09.532 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:16:09.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:09.533 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:16:10.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:10.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:10.577 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1742,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:16:10.607 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:10.607 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:16:11.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:11.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:11.592 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1742,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:16:11.614 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:11.614 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:16:12.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:12.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:12.613 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1742,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:16:12.640 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:16:12.641 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:12.641 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:16:13.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:13.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:13.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:16:14.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:14.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:14.800 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:16:14.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:14.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:15.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:15.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:15.783 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:15.814 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:15.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:16:15.814 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:16:15.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:15.815 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:16:16.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:16.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:16.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1742,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:16:16.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:16.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:16:17.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:17.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:17.841 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1742,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:16:17.864 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:17.864 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:16:18.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:18.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:18.839 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1742,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:16:18.869 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:16:18.869 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:16:18.869 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:16:18.869 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:16:18.869 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:16:18.870 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:16:18.888 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:16:18.888 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:16:18.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:18.889 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:19.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:19.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:20.122 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:20.153 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:20.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:20.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:21.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:21.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:22.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:22.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:23.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:23.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:23.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:23.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:24.144 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:24.166 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:24.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:24.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:25.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:25.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:26.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:26.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:27.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:27.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:27.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:27.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:28.123 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:28.145 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1742,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:28.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:28.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:29.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:29.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:30.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:30.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:31.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:31.147 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:16:31.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:31.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:32.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:32.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:33.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:33.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:33.900 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1744,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:33.930 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1744,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:16:33.931 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:16:33.931 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:33.931 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:16:34.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:34.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:34.929 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1744,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:16:34.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:16:34.962 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:16:35.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:35.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:35.993 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1744,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:16:36.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:16:36.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:36.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:37.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:37.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:38.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:38.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:39.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:39.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:40.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:40.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:41.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:41.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:42.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:42.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:43.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:43.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:44.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:44.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:45.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:45.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:46.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:46.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:47.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:47.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:48.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:48.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:49.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:49.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:50.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:50.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:51.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:51.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:52.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:52.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:53.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:53.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:54.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:54.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:55.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:55.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:56.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:56.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:57.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:57.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:58.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:58.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:16:59.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:16:59.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:00.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:00.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:01.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:01.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:02.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:02.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:03.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:03.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:04.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:04.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:05.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:05.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:06.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:06.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:07.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:07.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:08.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:08.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:09.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:09.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:10.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:10.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:11.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:11.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:12.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:12.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:13.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:13.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:14.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:14.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:15.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:15.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:16.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:16.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:17.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:17.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:18.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:18.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:19.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:19.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:20.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:20.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:21.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:21.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:22.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:22.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:23.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:23.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:24.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:24.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:25.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:25.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:26.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:26.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:27.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:27.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:28.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:28.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:29.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:29.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:30.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:30.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:31.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:31.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:32.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:32.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:33.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:33.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:34.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:34.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:35.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:35.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:36.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:36.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:37.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:37.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:38.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:38.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:39.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:39.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:40.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:40.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:41.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:41.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:42.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:42.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:43.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:43.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:44.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:44.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:45.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:45.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:46.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:46.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:47.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:47.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:48.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:48.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:49.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:49.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:50.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:50.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:51.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:51.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:52.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:52.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:53.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:53.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:54.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:54.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:55.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:55.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:56.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:56.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:57.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:57.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:58.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:58.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:17:59.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:17:59.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:00.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:00.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:01.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:01.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:02.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:02.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:03.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:03.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:04.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:04.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:05.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:05.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:06.021 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:18:06.021 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:18:06.021 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:18:06.022 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:18:06.022 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:18:06.028 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:18:06.028 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:18:06.028 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:06.028 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:06.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:06.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:07.056 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1744,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:07.077 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1744,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:07.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:07.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:08.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:08.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:09.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:09.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:10.078 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:10.078 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:10.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:10.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:11.083 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:11.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:11.115 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:18:11.115 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:11.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:18:11.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:11.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:12.124 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1748,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:18:12.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:12.147 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:18:12.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:12.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:13.147 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1748,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:18:13.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:13.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:18:13.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:13.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:14.147 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1748,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:18:14.169 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:18:14.169 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:14.169 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:18:14.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:14.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:15.278 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:18:15.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:15.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:16.279 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:18:16.280 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:16.280 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:16.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:16.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:17.257 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:17.286 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:17.286 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:18:17.286 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:18:17.287 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:17.287 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:18:17.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:17.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:18.296 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1748,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:18:18.327 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:18.328 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:18:18.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:18.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:19.355 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1748,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:18:19.388 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:19.388 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:18:19.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:19.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:20.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1748,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:18:20.427 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:18:20.428 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:18:20.428 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:18:20.428 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:18:20.428 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:18:20.428 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:18:20.465 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:18:20.465 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:18:20.465 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:20.465 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:20.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:20.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:21.529 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:21.558 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:21.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:21.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:22.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:22.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:23.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:23.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:24.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:24.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:24.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:24.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:25.577 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:25.600 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:25.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:25.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:26.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:26.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:27.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:27.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:28.601 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:28.601 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:28.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:28.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:29.594 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:29.615 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1748,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:29.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:29.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:30.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:30.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:31.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:31.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:32.617 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:32.617 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:18:32.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:32.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:33.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:33.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:34.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:34.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:35.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:35.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:35.636 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1752,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:35.668 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1752,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:18:35.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:18:35.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:35.669 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:18:36.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:36.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:36.662 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1752,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:18:36.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:18:36.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:18:37.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:37.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:37.682 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1752,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:18:37.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:18:38.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:38.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:39.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:39.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:40.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:40.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:41.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:41.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:42.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:42.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:43.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:43.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:44.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:44.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:45.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:45.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:46.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:46.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:47.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:47.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:48.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:48.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:49.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:49.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:50.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:50.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:51.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:51.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:52.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:52.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:53.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:53.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:54.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:54.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:55.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:55.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:56.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:56.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:57.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:57.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:58.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:58.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:18:59.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:18:59.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:00.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:00.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:01.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:01.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:02.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:02.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:03.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:03.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:04.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:04.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:05.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:05.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:06.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:06.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:07.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:07.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:08.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:08.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:09.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:09.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:10.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:10.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:11.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:11.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:12.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:12.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:13.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:13.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:14.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:14.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:15.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:15.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:16.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:16.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:17.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:17.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:18.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:18.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:19.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:19.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:20.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:20.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:21.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:21.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:22.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:22.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:23.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:23.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:24.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:24.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:25.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:25.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:26.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:26.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:27.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:27.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:28.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:28.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:29.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:29.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:30.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:30.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:31.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:31.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:32.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:32.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:33.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:33.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:34.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:34.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:35.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:35.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:36.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:36.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:37.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:37.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:38.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:38.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:39.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:39.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:40.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:40.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:41.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:41.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:42.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:42.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:43.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:43.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:44.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:44.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:45.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:45.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:46.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:46.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:47.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:47.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:48.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:48.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:49.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:49.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:50.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:50.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:51.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:51.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:52.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:52.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:53.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:53.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:54.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:54.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:55.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:55.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:56.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:56.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:57.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:57.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:58.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:58.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:19:59.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:19:59.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:00.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:00.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:01.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:01.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:02.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:02.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:03.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:03.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:04.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:04.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:05.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:05.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:06.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:06.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:07.713 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:20:07.713 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:20:07.713 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:20:07.713 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:07.714 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:20:07.720 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:20:07.720 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:20:07.720 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:07.720 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:07.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:07.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:08.728 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1752,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:08.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:08.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:08.757 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1752,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:09.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:09.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:10.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:10.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 130 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:11.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:11.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:11.758 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:11.758 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:12.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:12.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:13.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:13.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:14.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:14.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:15.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:15.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:16.684 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:16.723 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:16.724 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:20:16.724 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:16.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:20:16.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:16.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:17.693 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1758,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:20:17.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:17.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:17.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:17.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:20:18.731 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1758,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:20:18.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:18.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:18.753 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:18.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:20:19.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:19.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:19.753 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1758,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:20:19.776 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:20:19.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:19.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:20:20.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:20.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:20.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:20:21.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:21.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:21.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:20:21.895 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:21.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:22.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:22.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:22.893 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:22.917 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:22.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:20:22.917 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:20:22.917 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:22.918 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:20:23.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:23.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:23.897 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1758,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:20:23.926 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:23.926 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:20:24.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:24.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:24.937 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1758,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:20:24.958 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:24.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:20:25.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:25.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:25.948 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1758,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:20:25.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:20:25.978 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:20:25.978 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:20:25.978 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:20:25.979 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:20:25.979 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:20:26.008 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:20:26.008 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:20:26.009 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:26.009 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:26.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:26.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:27.068 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:27.090 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:27.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:27.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:28.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:28.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:29.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:29.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:30.091 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:30.091 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:30.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:30.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:31.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:31.103 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1758,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:31.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:31.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:32.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:32.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:33.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:33.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:34.104 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:34.105 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:20:34.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:34.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:35.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:35.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:36.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:36.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:37.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:37.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:38.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:38.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:39.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:39.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:40.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:40.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:41.206 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:41.237 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:20:41.237 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:20:41.238 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:41.238 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:20:41.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:41.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:42.250 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1762,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:20:42.283 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:20:42.284 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:20:42.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:42.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:43.304 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1762,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:20:43.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:20:43.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:43.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:44.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:44.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:45.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:45.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:46.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:46.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:47.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:47.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:48.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:48.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:49.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:49.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:50.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:50.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:51.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:51.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:52.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:52.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:53.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:53.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:54.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:54.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:55.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:55.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:56.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:56.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:57.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:57.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:58.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:58.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:20:59.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:20:59.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:00.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:00.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:01.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:01.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:02.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:02.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:03.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:03.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:04.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:04.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:05.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:05.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:06.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:06.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:07.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:07.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:08.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:08.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:09.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:09.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:10.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:10.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:11.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:11.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:12.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:12.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:13.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:13.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:14.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:14.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:15.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:15.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:16.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:16.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:17.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:17.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:18.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:18.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:19.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:19.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:20.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:20.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:21.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:21.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:22.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:22.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:23.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:23.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:24.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:24.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:25.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:25.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:26.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:26.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:27.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:27.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:28.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:28.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:29.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:29.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:30.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:30.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:31.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:31.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:32.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:32.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:33.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:33.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:34.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:34.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:35.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:35.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:36.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:36.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:37.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:37.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:38.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:38.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:39.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:39.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:40.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:40.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:41.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:41.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:42.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:42.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:43.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:43.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:44.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:44.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:45.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:45.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:46.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:46.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:47.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:47.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:49.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:49.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:50.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:50.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:51.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:51.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:52.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:52.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:53.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:53.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:54.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:54.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:55.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:55.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:56.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:56.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:57.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:57.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:58.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:58.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:21:59.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:21:59.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:00.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:00.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:01.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:01.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:02.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:02.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:03.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:03.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:04.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:04.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:05.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:05.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:06.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:06.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:07.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:07.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:08.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:08.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:09.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:09.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:10.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:10.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:11.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:11.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:12.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:12.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:12.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:12.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:13.328 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:22:13.328 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:22:13.328 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:22:13.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:13.329 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:22:13.342 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:22:13.343 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:22:13.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:13.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:14.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:14.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:14.317 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:14.336 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:15.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:15.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:16.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:16.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:17.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:17.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:17.338 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:17.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:18.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:18.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:18.348 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:18.371 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1762,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:19.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:19.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:20.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:20.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:21.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:21.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:21.372 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:21.372 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:22.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:22.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:22.404 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:22.434 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:22.435 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:22:22.435 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:22.435 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:22:23.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:23.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:23.469 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1764,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:23.492 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:23.492 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:22:24.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:24.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:24.493 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1764,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:22:24.524 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:24.525 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:22:25.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:25.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:25.510 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1764,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:25.540 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:22:25.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:25.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:22:26.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:26.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:26.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:22:27.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:27.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:27.665 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:22:27.665 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:27.665 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:28.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:28.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:28.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:28.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:28.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:22:28.669 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:22:28.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:28.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:22:28.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:28.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:29.692 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1764,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:29.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:29.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:22:29.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:29.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:30.725 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1764,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:22:30.748 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:30.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:22:30.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:30.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:31.762 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1764,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:31.784 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:22:31.785 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:22:31.785 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:22:31.785 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:22:31.785 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:22:31.786 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:22:31.804 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:22:31.804 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:22:31.804 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:31.804 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:31.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:31.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:32.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:32.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:32.852 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:32.875 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:33.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:33.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:34.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:34.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:35.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:35.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:35.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:35.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:36.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:36.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:36.866 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:36.888 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:37.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:37.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:38.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:38.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:39.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:39.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:39.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:39.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:40.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:40.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:40.886 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:40.909 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1764,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:41.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:41.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:42.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:42.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:43.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:43.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:43.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:43.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:22:44.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:44.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:45.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:45.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:46.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:46.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:47.251 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:47.274 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:22:47.274 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:22:47.275 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:47.275 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:22:47.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:47.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:48.279 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1768,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:48.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:22:48.302 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:22:48.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:48.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:49.303 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1768,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:22:49.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:22:49.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:49.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:50.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:50.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:51.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:51.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:52.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:52.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:53.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:53.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:54.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:54.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:55.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:55.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:56.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:56.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:57.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:57.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:58.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:58.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:22:59.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:22:59.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:00.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:00.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:01.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:01.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:02.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:02.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:03.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:03.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:04.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:04.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:05.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:05.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:06.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:06.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:07.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:07.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:08.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:08.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:09.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:09.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:10.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:10.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:11.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:11.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:12.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:12.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:13.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:13.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:14.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:14.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:15.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:15.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:16.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:16.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:17.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:17.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:18.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:18.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:19.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:19.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:20.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:20.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:21.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:21.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:22.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:22.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:23.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:23.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:24.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:24.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:25.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:25.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:26.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:26.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:27.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:27.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:28.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:28.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:29.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:29.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:30.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:30.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:31.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:31.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:32.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:32.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:33.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:33.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:34.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:34.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:35.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:35.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:36.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:36.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:37.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:37.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:38.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:38.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:39.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:39.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:40.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:40.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:41.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:41.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:42.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:42.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:43.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:43.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:44.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:44.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:45.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:45.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:46.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:46.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:47.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:47.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:48.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:48.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:49.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:49.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:50.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:50.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:51.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:51.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:52.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:52.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:53.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:53.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:54.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:54.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:55.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:55.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:56.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:56.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:57.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:57.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:58.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:58.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:23:59.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:23:59.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:00.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:00.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:01.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:01.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:02.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:02.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:03.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:03.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:04.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:04.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:05.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:05.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:06.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:06.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:07.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:07.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:08.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:08.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:09.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:09.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:10.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:10.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:11.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:11.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:12.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:12.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:13.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:13.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:14.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:14.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:15.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:15.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:16.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:16.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:17.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:17.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:18.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:18.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:19.326 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:24:19.326 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:24:19.326 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:24:19.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:19.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:24:19.332 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:24:19.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:24:19.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:19.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:19.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:19.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:20.329 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:20.347 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:20.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:20.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:21.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:21.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:22.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:22.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:23.348 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:23.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:23.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:23.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:24.359 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:24.385 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1768,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:24.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:24.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:25.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:25.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:26.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:26.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:27.387 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:27.387 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:27.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:27.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:28.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:28.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:29.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:29.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:30.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1772,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:30.058 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1772,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:30.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:30.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:31.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:31.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:32.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:32.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:33.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:33.059 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:33.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:33.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:34.063 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:34.086 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:34.086 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:24:34.087 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:34.087 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:24:34.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:34.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:35.021 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1774,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:35.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:35.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:24:35.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:35.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:36.027 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1774,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:36.050 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:36.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:24:36.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:36.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:37.053 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1774,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:37.076 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:24:37.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:37.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:24:37.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:37.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:38.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:24:39.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:38.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:39.138 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:24:39.139 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:39.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:40.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:40.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:40.118 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:40.140 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:40.141 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:24:40.141 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:24:40.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:40.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:24:41.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:41.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:41.142 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1774,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:41.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:41.164 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:24:42.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:42.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:42.132 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1774,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:42.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:42.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:24:43.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:43.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:43.170 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1774,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:43.200 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:24:43.201 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:24:43.201 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:24:43.201 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:24:43.201 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:24:43.202 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:24:43.219 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:24:43.219 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:24:43.220 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:43.220 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:44.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:44.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:44.279 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:44.300 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:45.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:45.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:46.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:46.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:47.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:47.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:47.300 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:47.300 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:48.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:48.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:48.275 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:48.298 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:49.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:49.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:50.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:50.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:51.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:51.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:51.301 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:51.301 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:52.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:52.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:52.309 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:52.332 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1774,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:53.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:53.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:54.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:54.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:55.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:55.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:55.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:55.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:24:56.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:56.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:57.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:57.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:58.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:58.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:58.526 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:58.561 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:24:58.561 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:24:58.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:58.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:24:59.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:24:59.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:24:59.562 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1778,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:24:59.598 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:24:59.598 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:25:00.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:00.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:00.601 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1778,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:25:00.634 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:25:01.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:01.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:02.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:02.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:03.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:03.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:04.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:04.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:05.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:05.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:06.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:06.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:07.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:07.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:08.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:08.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:09.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:09.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:10.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:10.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:11.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:11.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:12.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:12.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:13.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:13.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:14.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:14.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:15.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:15.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:16.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:16.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:17.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:17.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:18.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:18.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:19.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:19.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:20.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:20.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:21.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:21.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:22.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:22.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:23.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:23.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:24.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:24.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:25.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:25.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:26.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:26.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:27.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:27.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:28.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:28.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:29.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:29.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:30.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:30.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:31.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:31.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:32.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:32.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:33.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:33.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:34.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:34.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:35.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:35.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:36.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:36.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:37.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:37.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:38.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:38.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:39.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:39.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:40.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:40.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:41.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:41.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:42.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:42.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:43.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:43.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:44.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:44.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:45.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:45.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:46.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:46.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:47.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:47.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:48.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:48.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:49.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:49.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:50.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:50.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:51.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:51.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:52.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:52.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:53.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:53.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:54.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:54.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:55.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:55.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:56.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:56.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:57.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:57.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:58.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:58.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:25:59.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:25:59.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:00.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:00.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:01.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:01.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:02.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:02.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:03.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:03.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:04.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:04.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:05.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:05.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:06.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:06.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:07.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:07.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:08.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:08.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:09.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:09.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:10.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:10.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:11.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:11.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:12.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:12.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:13.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:13.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:14.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:14.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:15.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:15.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:16.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:16.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:17.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:17.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:18.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:18.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:19.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:19.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:20.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:20.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:21.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:21.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:22.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:22.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:23.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:23.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:24.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:24.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:25.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:25.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:26.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:26.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:27.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:27.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:28.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:28.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:29.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:29.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:30.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:30.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:30.636 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:26:30.636 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:26:30.636 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:26:30.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:30.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:26:30.646 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:26:30.647 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:26:30.647 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:30.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:26:31.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:31.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:31.633 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:31.655 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:32.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:32.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:33.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:33.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:34.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:34.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:34.656 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:34.656 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:26:35.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:35.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:35.665 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:35.695 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:36.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:36.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:37.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:37.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:38.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:38.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:38.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:38.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:26:39.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:39.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:39.779 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:39.802 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1778,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:40.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:40.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:41.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:41.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:42.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:42.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:42.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:42.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:26:43.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:43.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:43.777 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1780,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:43.814 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1780,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:43.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:26:43.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:43.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:26:44.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:44.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:44.820 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1780,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:44.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:44.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:26:45.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:45.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:45.851 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1780,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:45.879 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:45.880 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:26:46.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:46.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:46.893 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1780,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:46.917 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:26:46.917 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:46.917 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:26:47.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:47.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:48.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:26:48.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:48.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:49.041 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:26:49.041 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:49.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:26:49.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:49.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:50.043 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1780,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:50.062 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1780,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:26:50.063 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:26:50.063 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:26:50.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:50.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:26:50.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:50.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:51.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:51.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:51.092 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1780,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:51.115 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:51.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:26:52.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:52.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:52.118 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1780,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:52.142 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:26:52.142 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:26:53.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:53.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:53.150 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1780,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:26:53.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:26:53.173 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T09:26:53.173 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:26:53.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:26:53.174 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:26:53.174 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:26:53.187 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:26:53.187 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:26:53.188 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:26:53.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:26:54.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:54.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:55.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:55.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:56.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:56.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:57.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:57.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:58.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:58.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:26:59.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:26:59.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:00.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:00.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:01.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:01.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:02.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:02.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:03.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:03.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:03.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:03.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:04.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:04.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:05.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:05.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:06.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:06.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:07.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:07.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:08.189 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:27:08.190 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:27:08.190 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:27:08.191 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:27:08.191 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:27:08.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:08.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:09.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:09.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:10.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:10.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:11.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:11.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:12.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:12.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:13.836 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1784,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:27:13.866 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1784,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:27:13.866 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:27:13.866 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:27:13.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:27:13.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:13.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:14.838 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1784,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:27:14.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:27:14.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:27:14.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:14.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:15.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1784,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:27:15.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:27:16.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:16.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:17.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:17.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:17.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:17.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:18.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:18.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:19.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:19.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:20.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:20.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:21.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:21.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:22.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:22.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:23.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:23.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:24.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:24.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:25.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:25.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:26.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:26.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:27.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:27.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:28.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:28.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:29.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:29.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:30.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:30.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:31.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:31.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:32.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:32.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:33.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:33.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:34.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:34.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:35.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:35.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:36.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:36.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:38.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:38.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:39.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:39.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:40.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:40.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:41.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:41.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:42.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:42.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:43.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:43.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:44.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:44.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:45.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:45.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:46.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:46.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:46.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:46.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:47.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:47.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:48.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:48.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:50.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:50.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:50.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:50.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:52.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:52.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:53.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:53.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:54.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:54.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:55.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:55.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:56.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:56.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:57.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:57.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:58.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:57.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:58.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:58.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:27:59.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:27:59.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:00.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:00.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:01.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:01.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:02.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:02.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:03.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:03.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:05.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:05.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:05.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:05.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:06.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:06.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:07.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:07.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:08.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:08.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:09.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:09.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:10.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:10.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:11.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:11.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:12.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:12.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:13.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:13.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:14.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:14.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:16.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:16.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:16.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:16.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:17.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:17.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:18.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:18.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:19.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:19.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:20.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:20.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:21.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:21.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:22.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:22.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:23.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:23.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:24.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:24.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:25.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:25.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:26.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:26.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:27.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:27.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:28.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:28.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:29.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:29.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:30.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:30.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:31.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:31.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:32.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:32.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:33.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:33.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:34.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:34.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:35.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:35.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:36.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:36.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:37.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:37.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:38.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:38.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:39.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:39.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:40.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:40.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:41.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:41.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:42.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:42.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:43.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:43.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:44.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:44.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:45.860 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:28:45.860 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:28:45.860 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:28:45.861 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:28:45.861 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:28:45.867 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:28:45.867 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:28:45.868 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:28:45.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:28:45.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:45.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:47.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:47.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:48.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:48.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:49.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:49.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:49.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:49.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:50.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:50.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:52.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:52.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:52.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:52.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:53.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:53.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:54.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:54.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:55.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:55.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:56.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:56.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:57.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:57.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:28:59.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:28:59.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:00.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:00.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:00.869 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:29:00.869 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:29:00.869 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:29:00.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:00.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:01.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:01.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:01.869 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:01.900 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:01.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:29:01.901 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:01.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:29:02.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:02.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:02.906 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1786,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:02.933 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:02.933 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:29:03.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:03.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:03.933 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1786,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:03.964 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:03.964 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:29:04.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:04.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:04.957 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1786,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:04.975 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:29:04.975 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:04.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:29:05.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:05.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:06.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:06.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:06.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:29:07.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:07.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:07.122 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:29:07.122 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:07.122 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:08.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:08.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:08.123 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:08.145 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:08.146 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:29:08.146 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:29:08.146 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:08.146 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:29:09.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:09.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:09.129 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1786,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:09.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:09.148 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:29:10.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:10.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:10.143 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1786,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:10.166 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:10.166 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:29:11.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:11.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:11.224 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1786,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:11.247 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:29:11.248 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:29:11.248 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:29:11.248 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:29:11.248 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:29:11.248 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:29:11.284 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:29:11.284 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:29:11.284 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:11.284 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:12.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:12.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:12.280 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:12.302 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:13.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:12.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:13.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:13.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:14.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:14.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:15.304 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:15.304 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:15.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:15.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:16.319 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:16.341 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:17.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:17.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:17.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:17.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:19.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:19.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:19.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:19.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:20.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:20.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:20.358 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:20.381 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1786,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:21.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:21.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:22.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:22.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:23.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:23.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:23.381 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:23.382 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:29:24.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:24.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:25.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:25.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:26.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:26.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:27.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:27.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:27.425 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:27.448 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:29:27.448 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:29:27.448 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:27.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:29:28.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:28.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:28.474 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1788,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:29:28.496 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:29:28.496 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:29:29.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:29.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:29.517 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1788,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:29:29.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:29:30.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:30.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:31.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:31.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:32.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:32.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:33.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:33.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:34.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:34.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:35.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:35.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:36.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:36.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:37.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:37.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:38.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:38.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:39.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:39.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:40.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:40.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:41.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:41.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:42.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:42.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:43.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:43.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:44.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:44.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:45.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:45.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:46.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:46.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:47.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:47.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:48.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:48.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:49.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:49.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:50.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:50.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:51.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:51.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:52.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:52.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:53.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:53.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:54.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:54.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:55.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:55.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:56.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:56.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:57.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:57.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:58.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:58.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:29:59.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:29:59.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:00.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:00.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:01.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:01.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:02.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:02.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:03.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:03.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:04.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:04.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:05.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:05.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:06.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:06.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:07.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:07.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:08.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:08.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:09.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:09.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:10.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:10.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:11.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:11.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:12.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:12.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:13.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:13.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:14.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:14.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:15.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:15.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:16.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:16.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:17.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:17.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:18.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:18.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:19.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:19.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:20.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:20.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:21.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:21.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:22.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:22.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:23.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:23.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:24.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:24.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:25.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:25.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:26.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:26.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:27.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:27.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:28.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:28.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:29.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:29.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:30.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:30.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:31.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:31.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:32.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:32.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:33.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:33.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:34.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:34.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:35.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:35.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:36.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:36.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:37.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:37.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:38.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:38.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:39.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:39.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:40.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:40.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:41.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:41.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:42.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:42.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:43.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:43.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:44.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:44.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:45.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:45.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:46.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:46.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:47.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:47.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:48.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:48.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:49.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:49.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:50.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:50.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:51.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:51.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:52.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:52.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:53.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:53.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:54.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:54.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:55.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:55.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:56.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:56.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:57.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:57.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:58.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:58.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:59.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:30:59.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:30:59.549 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:30:59.549 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:30:59.549 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:30:59.549 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:30:59.550 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:30:59.561 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:30:59.561 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:30:59.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:30:59.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:00.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:00.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:00.576 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:00.598 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:01.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:01.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:02.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:02.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:03.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:03.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:03.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:03.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:04.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:04.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:04.605 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:04.637 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:05.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:05.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:06.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:06.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:07.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:07.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:07.639 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:07.639 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:08.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:08.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:08.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:08.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:09.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:09.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:10.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:10.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:11.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:11.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:11.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:11.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:12.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:12.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:12.756 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:12.781 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1788,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:13.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:13.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:14.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:14.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:15.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:15.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:15.782 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:15.782 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:16.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:16.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:16.784 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:16.806 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:16.807 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:31:16.807 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:16.807 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:31:17.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:17.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:17.770 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1792,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:31:17.793 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:17.793 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:31:18.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:18.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:18.768 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1792,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:18.790 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:18.790 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:31:19.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:19.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:19.797 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1792,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:19.820 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:31:19.821 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:19.821 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:31:20.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:20.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:20.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:31:21.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:21.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:21.765 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:31:21.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:21.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:22.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:22.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:22.761 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:22.782 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:22.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:31:22.783 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:31:22.783 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:22.783 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:31:23.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:23.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:23.776 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1792,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:31:23.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:23.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:31:24.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:24.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:24.834 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1792,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:24.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:24.857 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:31:25.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:25.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:25.875 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1792,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:25.897 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:31:25.897 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:31:25.897 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:31:25.898 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:31:25.898 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:31:25.898 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:31:25.918 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:31:25.918 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:31:25.918 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:25.918 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:26.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:26.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:26.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:27.011 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:27.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:27.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:28.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:28.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:29.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:29.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:30.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:30.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:30.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:30.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:31.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:31.050 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:31.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:31.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:32.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:32.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:33.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:33.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:34.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:34.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:34.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:34.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:35.060 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:35.083 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1792,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:35.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:35.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:36.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:36.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:37.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:37.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:38.084 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:38.084 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:31:38.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:38.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:39.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:39.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:40.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:40.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:41.243 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1796,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:41.274 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1796,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:31:41.275 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:31:41.275 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:41.275 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:31:41.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:41.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:42.275 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1796,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:42.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:31:42.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:31:42.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:42.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:43.288 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1796,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:31:43.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:31:43.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:43.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:44.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:44.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:45.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:45.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:46.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:46.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:47.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:47.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:48.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:48.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:49.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:49.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:50.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:50.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:51.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:51.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:52.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:52.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:53.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:53.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:54.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:54.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:55.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:55.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:56.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:56.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:57.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:57.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:58.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:58.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:31:59.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:31:59.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:00.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:00.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:01.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:01.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:02.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:02.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:03.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:03.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:04.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:04.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:05.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:05.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:06.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:06.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:07.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:07.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:08.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:08.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:09.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:09.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:10.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:10.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:11.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:11.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:12.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:12.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:13.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:13.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:14.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:14.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:15.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:15.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:16.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:16.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:17.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:17.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:18.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:18.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:19.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:19.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:20.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:20.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:21.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:21.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:22.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:22.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:23.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:23.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:24.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:24.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:25.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:25.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:26.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:26.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:27.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:27.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:28.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:28.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:29.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:29.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:30.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:30.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:31.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:31.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:32.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:32.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:33.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:33.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:34.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:34.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:35.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:35.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:36.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:36.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:37.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:37.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:38.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:38.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:39.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:39.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:40.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:40.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:41.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:41.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:42.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:42.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:43.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:43.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:44.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:44.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:45.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:45.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:46.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:46.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:47.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:47.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:48.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:48.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:49.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:49.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:50.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:50.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:51.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:51.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:52.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:52.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:53.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:53.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:54.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:54.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:55.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:55.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:56.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:56.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:57.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:57.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:58.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:58.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:32:59.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:32:59.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:00.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:00.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:01.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:01.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:02.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:02.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:03.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:03.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:04.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:04.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:05.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:05.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:06.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:06.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:07.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:07.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:08.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:08.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:09.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:09.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:10.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:10.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:11.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:11.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:12.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:12.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:13.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:13.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:13.312 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:33:13.312 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:33:13.312 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:33:13.313 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:13.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:33:13.318 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:33:13.318 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:33:13.319 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:13.319 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:14.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:14.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:14.303 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1796,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:14.325 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1796,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:15.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:15.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:16.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:16.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:17.327 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:17.327 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:17.354 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:17.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:18.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:18.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:19.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:19.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:20.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:20.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:21.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:21.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:22.196 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:22.219 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:22.219 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:33:22.219 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:22.219 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:33:22.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:22.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:23.276 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1802,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:33:23.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:23.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:33:23.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:23.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:24.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:24.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:24.529 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1802,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:24.562 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:24.562 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:33:25.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:25.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:25.670 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1802,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:25.694 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:33:25.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:25.695 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:33:26.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:26.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:26.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:33:27.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:27.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:27.850 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:33:27.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:27.851 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:28.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:28.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:28.850 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:28.881 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:28.881 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:33:28.881 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:33:28.882 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:28.882 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:33:29.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:29.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:29.881 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1802,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:33:29.909 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:29.909 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:33:30.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:30.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:30.917 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1802,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:30.949 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:30.949 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:33:31.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:31.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:31.970 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1802,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:32.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:33:32.000 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:33:32.000 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:33:32.000 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:33:32.001 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:33:32.001 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:33:32.024 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:33:32.024 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:33:32.025 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:32.025 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:32.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:32.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:33.032 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:33.065 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:33.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:33.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:34.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:34.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:35.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:35.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:36.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:36.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:36.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:36.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:37.061 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:37.082 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:37.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:37.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:38.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:38.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:39.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:39.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:40.083 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:40.084 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:40.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:40.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:41.068 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:41.098 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1802,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:41.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:41.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:42.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:42.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:43.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:43.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:44.100 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:44.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:33:44.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:44.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:45.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:45.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:46.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:46.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:47.243 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:47.266 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:33:47.266 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:33:47.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:47.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:33:47.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:47.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:48.233 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1804,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:48.254 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:33:48.255 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:33:48.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:48.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:49.215 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1804,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:33:49.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:33:49.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:49.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:50.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:50.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:51.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:51.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:52.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:52.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:53.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:53.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:54.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:54.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:55.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:55.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:56.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:56.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:57.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:57.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:58.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:58.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:33:59.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:33:59.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:00.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:00.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:01.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:01.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:02.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:02.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:03.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:03.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:04.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:04.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:05.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:05.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:06.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:06.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:07.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:07.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:08.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:08.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:09.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:09.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:10.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:10.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:11.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:11.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:12.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:12.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:13.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:13.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:14.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:14.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:15.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:15.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:16.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:16.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:17.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:17.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:18.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:18.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:19.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:19.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:20.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:20.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:21.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:21.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:22.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:22.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:23.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:23.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:24.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:24.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:25.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:25.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:26.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:26.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:27.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:27.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:28.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:28.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:29.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:29.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:30.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:30.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:31.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:31.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:32.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:32.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:33.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:33.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:34.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:34.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:35.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:35.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:36.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:36.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:37.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:37.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:38.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:38.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:39.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:39.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:40.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:40.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:41.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:41.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:42.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:42.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:43.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:43.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:44.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:44.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:45.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:45.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:46.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:46.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:47.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:47.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:48.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:48.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:49.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:49.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:50.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:50.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:51.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:51.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:52.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:52.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:53.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:53.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:54.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:54.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:55.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:55.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:56.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:56.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:57.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:57.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:58.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:58.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:34:59.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:34:59.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:00.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:00.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:01.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:01.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:02.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:02.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:03.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:03.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:04.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:04.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:05.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:05.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:06.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:06.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:07.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:07.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:08.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:08.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:09.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:09.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:10.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:10.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 135 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:11.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:11.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:12.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:12.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:13.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:13.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:14.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:14.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:15.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:15.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:16.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:16.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:17.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:17.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:18.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:18.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:19.243 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:35:19.243 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:35:19.243 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:35:19.244 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:35:19.244 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:35:19.275 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:35:19.275 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:35:19.276 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:19.276 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:19.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:19.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:20.305 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:20.331 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:20.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:20.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:21.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:21.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:22.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:22.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:23.331 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:23.332 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:23.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:23.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:24.348 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:24.370 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:24.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:24.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:25.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:25.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:26.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:26.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:27.376 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:27.376 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:27.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:27.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:28.344 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:28.366 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:28.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:28.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:29.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:29.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:30.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:30.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:31.367 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:31.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:31.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:31.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:32.367 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:32.387 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1804,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:32.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:32.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:33.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:33.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:34.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:34.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:35.393 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:35.393 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:35.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:35.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:36.407 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1808,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:36.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:36.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:36.437 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1808,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:36.437 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:35:36.438 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:36.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:35:37.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:37.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:37.509 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1808,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:37.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:37.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:35:38.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:38.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:38.536 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1808,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:38.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:38.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:35:39.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:39.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:39.560 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1808,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:39.582 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:35:39.582 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:39.583 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:35:40.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:40.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:40.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:35:41.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:41.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:41.668 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:35:41.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:41.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:35:42.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:42.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:42.672 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1808,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:42.704 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1808,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:35:42.704 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:35:42.704 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:35:42.705 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:42.705 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:35:43.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:43.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:43.708 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1808,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:43.730 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:43.730 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:35:44.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:44.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:44.824 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1808,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:44.847 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:35:44.847 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:35:45.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:45.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:45.836 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1808,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:35:45.859 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:35:45.859 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T09:35:45.859 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:35:45.859 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:35:45.860 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:35:45.860 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:35:45.879 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:35:45.879 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:35:45.880 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:35:45.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:35:46.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:46.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:47.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:47.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:48.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:48.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:49.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:49.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:50.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:50.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:51.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:51.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:52.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:52.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:53.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:53.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:54.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:54.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:55.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:55.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:56.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:56.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:57.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:57.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:58.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:58.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:35:59.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:35:59.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:00.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:00.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:00.881 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:36:00.881 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:36:00.882 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:36:00.882 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:36:00.882 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:36:01.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:01.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:02.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:02.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:03.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:03.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:04.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:04.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:05.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:05.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:06.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:06.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:06.545 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1810,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:36:06.574 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1810,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:36:06.574 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:36:06.575 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:36:06.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:36:07.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:07.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:07.708 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1810,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:36:07.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:36:07.739 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:36:08.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:08.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:08.735 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1810,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:36:08.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:36:09.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:09.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:10.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:10.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:11.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:11.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:12.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:12.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:13.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:13.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:14.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:14.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:15.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:15.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:16.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:16.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:17.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:17.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:18.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:18.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:19.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:19.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:20.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:20.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:21.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:21.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:22.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:22.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:23.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:23.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:24.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:24.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:25.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:25.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:26.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:26.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:27.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:27.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:28.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:28.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:29.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:29.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:30.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:30.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:31.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:31.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:32.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:32.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:33.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:33.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:34.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:34.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:35.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:35.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:36.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:36.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:37.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:37.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:38.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:38.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:39.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:39.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:40.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:40.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:41.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:41.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:42.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:42.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:43.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:43.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:44.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:44.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:45.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:45.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:46.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:46.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:47.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:47.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:48.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:48.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:49.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:49.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:50.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:50.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:51.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:51.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:52.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:52.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:53.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:53.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:54.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:54.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:55.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:55.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:56.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:56.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:57.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:57.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:58.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:58.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:36:59.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:36:59.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:00.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:00.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:01.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:01.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:02.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:02.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:03.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:03.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:04.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:04.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:05.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:05.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:06.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:06.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:07.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:07.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:08.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:08.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:09.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:09.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:10.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:10.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:11.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:11.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:12.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:12.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:13.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:13.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:14.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:14.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:15.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:15.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:16.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:16.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:17.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:17.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:18.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:18.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:19.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:19.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:20.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:20.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:21.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:21.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:22.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:22.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:23.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:23.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:24.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:24.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:25.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:25.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:26.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:26.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:27.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:27.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:28.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:28.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:29.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:29.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:30.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:30.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:31.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:31.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:32.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:32.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:33.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:33.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:34.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:34.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:35.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:35.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:36.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:36.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:36.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:36.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:38.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:38.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:38.758 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:37:38.758 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:37:38.758 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:37:38.759 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:38.759 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:37:38.765 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:37:38.765 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T09:37:38.766 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T09:37:38.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:37:39.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:39.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:40.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:40.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:41.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:41.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:42.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:42.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:43.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:43.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:44.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:44.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:45.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:45.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:46.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:46.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:47.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:47.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:48.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:48.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:49.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:49.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:50.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:50.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:51.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:51.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:52.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:52.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:53.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:53.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:53.766 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T09:37:53.767 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T09:37:53.767 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:37:53.767 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:53.768 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:37:54.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:54.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:54.782 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:37:54.804 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:37:54.804 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:37:54.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:54.805 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:37:55.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:55.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:55.807 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1814,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:37:55.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:55.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:37:56.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:56.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:56.766 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1814,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:37:56.788 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:56.789 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:37:57.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:57.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:57.769 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1814,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:37:57.799 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:37:57.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:57.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:37:58.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:58.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:58.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:37:59.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:37:59.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:37:59.928 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:37:59.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:37:59.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:38:00.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:00.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:00.905 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:00.935 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:00.935 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:38:00.936 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:38:00.936 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:00.936 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:38:00.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:00.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:01.918 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1814,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:38:01.937 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:01.937 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:38:01.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:01.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:03.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:02.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:03.013 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1814,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:38:03.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:03.036 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:38:04.011 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1814,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:38:04.033 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:38:04.033 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:38:04.033 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:38:04.034 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:38:04.034 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:38:04.034 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:38:04.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:04.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:04.053 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:38:04.053 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:38:04.053 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:04.053 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:38:05.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:05.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:05.101 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:05.124 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:06.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:06.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:07.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:07.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:07.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:07.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:08.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:08.125 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:38:08.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:08.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:09.106 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:09.132 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1814,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:10.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:09.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:11.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:11.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:12.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:12.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:12.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:12.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:38:13.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:13.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:14.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:14.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:15.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:15.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:16.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:16.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:17.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:17.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:18.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:18.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:19.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:19.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:19.571 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1818,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:19.593 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1818,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:38:19.593 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:38:19.594 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:19.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:38:20.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:20.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:20.604 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1818,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:38:20.629 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:38:20.629 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:38:21.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:21.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:21.618 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1818,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:38:21.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:38:22.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:22.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:23.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:23.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:24.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:24.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:24.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:24.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:26.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:26.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:26.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:26.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:27.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:27.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:28.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:28.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:29.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:29.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:30.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:30.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:31.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:31.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:32.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:32.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:33.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:33.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:34.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:34.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:35.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:35.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:36.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:36.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:37.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:37.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:38.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:38.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:39.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:39.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:40.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:40.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:41.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:41.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:43.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:43.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:44.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:43.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:45.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:45.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:46.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:46.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:47.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:47.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:48.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:48.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:49.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:49.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:50.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:50.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:51.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:51.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:52.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:52.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:53.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:53.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:54.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:54.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:55.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:55.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:56.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:56.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:57.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:57.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:58.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:58.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:38:59.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:38:59.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:00.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:00.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:01.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:01.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:02.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:02.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:03.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:03.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:04.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:04.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:05.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:05.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:06.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:06.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:07.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:07.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:08.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:08.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:09.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:09.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:10.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:10.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:11.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:11.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:12.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:12.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:13.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:13.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:14.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:14.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:15.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:15.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:16.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:16.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:17.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:17.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:18.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:18.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:19.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:19.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:20.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:20.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:21.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:21.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:22.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:22.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:23.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:23.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:24.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:24.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:25.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:25.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:26.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:26.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:27.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:27.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:28.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:28.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:29.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:29.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:30.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:30.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:31.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:31.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:32.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:32.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:33.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:33.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:34.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:34.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:35.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:35.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:36.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:36.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:37.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:37.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:38.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:38.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:39.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:39.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:40.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:40.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:41.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:41.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:42.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:42.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:43.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:43.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:44.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:44.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:45.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:45.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:46.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:46.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:47.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:47.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:48.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:48.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:49.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:49.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:50.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:50.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:51.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:51.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:51.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:39:51.642 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:39:51.642 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:39:51.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:51.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:39:51.649 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:39:51.650 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:39:51.650 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:51.650 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:39:52.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:52.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:52.639 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1818,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:39:52.660 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1818,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:39:53.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:53.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:54.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:54.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:55.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:55.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:55.661 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:55.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:39:56.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:56.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:56.624 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1820,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:39:56.657 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1820,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:39:56.657 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:39:56.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:56.657 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:39:57.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:57.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:57.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1820,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:39:57.647 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:57.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:39:58.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:58.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:58.631 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1820,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:39:58.663 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:58.663 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:39:59.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:39:59.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:39:59.665 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1820,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:39:59.687 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:39:59.687 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:39:59.688 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:40:00.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:00.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:00.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:40:01.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:01.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:01.784 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:40:01.785 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:01.785 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:40:02.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:02.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:02.792 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1820,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:40:02.824 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1820,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:40:02.825 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:40:02.825 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:40:02.825 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:02.825 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:40:03.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:03.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:03.837 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1820,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:40:03.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:03.871 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:40:04.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:04.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:04.892 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1820,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:40:04.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:04.924 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:40:05.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:05.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:05.932 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1820,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:40:05.954 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:40:05.955 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T09:40:05.955 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:40:05.955 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:40:05.956 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:40:05.956 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:40:05.984 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:40:05.984 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T09:40:05.985 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T09:40:05.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:40:06.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:06.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:07.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:07.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:08.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:08.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:09.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:09.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:10.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:10.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:11.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:11.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:12.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:12.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:13.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:13.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:14.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:14.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:15.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:15.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:16.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:16.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:17.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:17.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:18.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:18.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:19.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:19.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:20.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:20.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:20.985 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T09:40:20.986 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T09:40:20.986 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:40:20.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:20.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:40:21.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:21.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:22.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:22.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:23.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:23.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:24.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:24.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:25.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:25.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:26.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:26.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:26.614 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1822,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:40:26.636 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1822,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:40:26.637 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:40:26.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:26.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:40:27.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:27.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:27.658 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1822,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:40:27.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:40:27.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:40:28.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:28.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:28.668 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1822,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:40:28.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:40:29.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:29.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:30.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:30.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:31.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:31.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:32.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:32.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:33.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:33.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:34.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:34.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:35.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:35.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:36.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:36.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:37.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:37.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:38.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:38.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:39.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:39.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:40.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:40.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:41.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:41.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:42.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:42.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:43.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:43.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:44.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:44.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:45.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:45.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:46.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:46.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:47.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:47.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:48.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:48.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:49.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:49.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:50.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:50.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:51.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:51.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:52.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:52.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:53.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:53.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:54.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:54.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:55.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:55.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:56.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:56.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:57.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:57.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:58.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:58.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:40:59.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:40:59.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:00.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:00.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:01.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:01.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:02.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:02.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:03.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:03.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:04.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:04.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:05.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:05.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:06.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:06.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:07.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:07.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:08.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:08.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:09.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:09.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:10.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:10.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:11.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:11.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:12.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:12.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:13.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:13.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:14.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:14.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:15.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:15.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:16.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:16.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:17.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:17.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:18.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:18.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:19.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:19.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:20.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:20.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:21.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:21.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:22.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:22.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:23.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:23.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:24.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:24.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:25.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:25.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:26.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:26.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:27.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:27.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:28.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:28.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:29.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:29.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:30.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:30.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:31.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:31.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:32.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:32.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:33.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:33.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:34.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:34.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:35.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:35.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:36.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:36.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:37.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:37.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:38.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:38.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:39.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:39.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:40.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:40.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:41.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:41.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:42.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:42.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:43.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:43.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:44.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:44.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:45.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:45.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:46.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:46.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:47.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:47.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:48.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:48.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:49.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:49.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:50.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:50.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:51.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:51.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:52.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:52.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:53.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:53.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:54.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:54.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:55.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:55.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:56.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:56.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:57.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:57.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:58.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:58.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:41:58.691 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:41:58.691 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:41:58.692 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:41:58.692 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:41:58.692 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:41:58.698 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:41:58.698 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T09:41:58.698 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T09:41:58.698 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:41:59.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:41:59.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:00.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:00.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:01.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:01.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:02.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:02.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:03.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:03.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:04.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:04.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:05.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:05.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:06.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:06.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:07.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:07.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:08.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:08.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:09.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:09.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:10.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:10.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:11.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:11.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:12.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:12.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:13.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:13.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:13.699 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T09:42:13.699 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T09:42:13.699 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:42:13.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:13.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:42:14.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:14.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:14.722 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1830,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:14.744 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1830,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:14.745 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:42:14.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:14.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:42:15.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:15.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:15.745 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1830,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:15.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:15.763 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:42:16.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:16.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:16.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1830,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:16.768 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:16.769 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:42:17.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:17.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:17.769 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1830,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:17.791 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:42:17.791 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:17.791 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:42:18.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:18.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:18.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:42:19.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:19.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:19.873 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:42:19.873 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:19.873 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:42:20.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:20.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:20.884 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1830,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:20.914 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1830,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:20.914 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:42:20.915 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:42:20.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:20.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:42:21.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:21.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:21.916 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1830,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:21.937 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:21.937 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:42:22.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:22.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:22.837 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1830,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:22.859 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:22.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:42:23.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:23.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:23.910 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1830,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:23.928 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:42:23.929 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T09:42:23.929 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:42:23.929 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:42:23.929 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:42:23.929 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:42:23.950 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:42:23.950 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:42:23.950 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:42:23.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:42:24.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:24.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:25.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:25.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:26.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:26.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:27.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:27.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:28.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:28.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:29.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:29.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:30.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:30.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:31.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:31.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:32.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:32.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:33.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:33.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:34.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:34.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:35.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:35.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:36.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:36.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:37.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:37.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:38.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:38.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:38.961 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:42:38.961 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:42:38.961 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:42:38.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:38.962 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:42:39.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:39.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:40.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:40.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:41.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:41.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:42.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:42.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:43.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:43.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:44.620 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1834,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:44.651 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1834,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:42:44.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:42:44.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:44.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:42:44.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:44.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:45.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1834,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:42:45.612 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:42:45.612 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:42:45.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:45.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:46.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:46.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:46.613 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1834,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:42:46.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:42:47.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:47.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:48.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:48.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:49.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:49.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:50.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:50.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:51.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:51.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:52.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:52.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:53.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:53.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:54.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:54.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:55.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:55.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:56.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:56.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:57.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:57.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:58.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:58.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:42:59.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:42:59.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:00.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:00.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:01.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:01.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:02.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:02.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:03.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:03.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:04.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:04.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:05.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:05.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:06.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:06.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:07.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:07.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:08.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:08.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:09.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:09.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:10.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:10.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:11.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:11.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:12.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:12.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:13.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:13.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:14.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:14.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:15.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:15.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:16.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:16.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:17.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:17.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:18.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:18.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:19.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:19.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:20.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:20.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:21.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:21.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:22.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:22.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:23.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:23.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:24.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:24.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:25.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:25.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:26.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:26.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:27.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:27.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:28.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:28.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:29.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:29.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:30.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:30.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:31.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:31.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:32.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:32.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:33.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:33.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:34.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:34.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:35.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:35.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:36.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:36.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:37.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:37.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:38.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:38.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:39.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:39.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:40.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:40.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:41.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:41.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:42.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:42.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:43.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:43.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:44.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:44.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:45.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:45.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:46.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:46.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:47.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:47.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:48.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:48.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:49.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:49.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:50.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:50.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:51.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:51.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:52.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:52.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:53.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:53.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:54.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:54.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:55.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:55.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:56.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:56.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:57.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:57.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:58.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:58.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:43:59.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:43:59.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:00.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:00.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:01.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:01.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:02.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:02.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:03.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:03.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:04.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:04.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:05.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:05.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:06.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:06.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:07.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:07.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:08.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:08.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:09.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:09.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:10.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:10.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:11.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:11.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:12.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:12.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:13.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:13.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:14.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:14.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:15.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:15.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:16.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:16.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:16.636 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:44:16.636 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:44:16.636 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:44:16.636 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:16.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:44:16.647 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:44:16.647 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T09:44:16.647 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T09:44:16.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T09:44:17.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:17.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:18.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:18.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:19.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:19.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:20.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:20.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:21.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:21.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:22.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:22.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:23.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:23.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:24.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:24.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:25.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:25.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:26.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:26.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:27.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:27.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:28.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:28.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:29.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:29.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:30.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:30.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:31.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:31.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:31.648 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T09:44:31.649 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T09:44:31.649 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:44:31.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:31.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:32.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:32.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:32.656 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:32.680 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:32.680 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:44:32.681 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:32.681 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:44:33.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:33.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:33.663 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1838,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:44:33.690 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:33.690 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:44:34.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:34.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:34.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1838,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:44:34.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:34.710 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:44:35.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:35.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:35.728 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1838,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:44:35.752 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:44:35.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:35.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:44:36.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:36.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:36.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:44:37.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:37.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:37.909 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:44:37.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:37.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:38.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:38.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:38.928 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:38.960 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:38.960 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:44:38.961 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:44:38.961 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:38.961 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:44:39.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:39.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:39.947 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1838,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:44:39.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:39.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:44:40.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:40.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:40.958 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1838,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:44:40.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:40.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:44:41.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:41.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:42.008 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1838,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:44:42.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:44:42.030 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:44:42.030 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:44:42.031 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:44:42.031 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:44:42.031 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:44:42.072 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:44:42.072 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:44:42.072 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:42.072 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:42.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:42.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:43.100 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:43.131 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:43.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:43.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:44.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:44.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:45.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:45.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:46.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:46.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:46.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:46.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:47.150 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:47.173 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:47.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:47.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:48.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:48.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:49.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:49.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:50.174 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:50.174 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:50.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:50.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:51.172 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:51.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1838,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:51.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:51.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:52.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:52.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:53.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:53.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:54.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:54.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:44:54.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:54.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:55.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:55.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:56.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:56.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:57.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:57.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:58.262 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:58.285 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:44:58.285 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:44:58.285 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:58.285 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:44:58.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:58.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:44:59.295 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1842,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:44:59.317 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:44:59.317 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:44:59.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:44:59.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:00.310 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1842,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:45:00.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:45:00.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:00.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:01.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:01.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:02.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:02.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:03.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:03.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:04.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:04.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:05.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:05.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:06.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:06.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:07.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:07.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:08.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:08.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:09.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:09.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:10.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:10.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:11.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:11.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:12.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:12.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:13.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:13.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:14.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:14.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:15.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:15.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:16.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:16.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:17.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:17.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:18.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:18.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:19.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:19.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:20.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:20.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:21.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:21.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:22.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:22.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:23.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:23.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:24.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:24.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:25.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:25.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:26.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:26.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:27.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:27.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:28.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:28.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:29.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:29.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:30.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:30.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:31.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:31.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:32.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:32.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:33.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:33.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:34.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:34.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:35.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:35.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:36.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:36.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:37.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:37.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:38.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:38.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:39.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:39.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:40.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:40.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:41.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:41.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:42.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:42.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:43.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:43.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:44.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:44.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:45.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:45.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:46.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:46.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:47.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:47.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:48.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:48.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:49.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:49.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:50.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:50.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:51.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:51.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:52.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:52.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:53.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:53.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:54.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:54.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:55.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:55.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:56.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:56.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:57.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:57.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:58.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:58.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:45:59.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:45:59.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:00.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:00.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:01.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:01.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:02.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:02.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:03.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:03.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:04.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:04.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:05.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:05.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:06.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:06.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:07.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:07.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:08.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:08.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:09.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:09.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:10.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:10.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:11.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:11.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:12.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:12.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:13.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:13.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:14.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:14.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:15.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:15.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:16.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:16.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:17.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:17.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:18.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:18.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:19.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:19.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:20.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:20.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:21.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:21.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:22.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:22.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:23.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:23.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:24.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:24.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:25.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:25.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:26.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:26.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:27.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:27.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:28.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:28.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:29.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:29.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:30.332 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:46:30.332 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:46:30.333 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:46:30.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:30.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:46:30.344 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:46:30.344 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:46:30.345 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:30.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:30.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:30.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:31.331 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:31.363 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:31.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:31.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:32.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:32.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:33.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:33.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:34.364 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:34.364 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:34.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:34.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:35.368 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:35.391 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1842,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:35.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:35.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:36.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:36.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:37.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:37.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:38.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:38.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:38.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:38.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:39.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:39.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:40.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:40.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:41.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:41.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:42.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:42.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:43.090 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:43.120 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:43.120 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:46:43.121 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:43.121 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:46:44.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:44.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:44.128 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1850,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:44.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:44.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:46:45.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:45.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:45.129 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1850,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:45.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:45.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:46:46.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:46.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:46.141 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1850,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:46.164 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:46:46.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:46.164 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:46:47.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:47.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:47.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:46:48.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:47.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:48.311 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:46:48.311 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:48.311 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:48.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:48.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:49.301 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:49.323 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:49.324 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:46:49.324 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:46:49.324 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:49.324 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:46:49.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:49.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:50.305 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1850,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:50.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:50.329 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:46:50.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:50.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:51.290 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1850,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:51.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:51.313 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:46:51.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:51.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:52.316 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1850,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:46:52.346 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:46:52.347 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:46:52.347 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:46:52.347 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:46:52.347 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:46:52.347 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:46:52.365 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:46:52.366 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:46:52.366 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:52.366 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:52.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:52.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:53.422 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:53.444 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:53.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:53.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:54.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:54.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:55.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:55.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:56.446 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:46:56.446 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:46:56.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:56.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:57.443 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:57.466 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:46:57.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:57.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:58.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:58.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:46:59.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:46:59.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:00.467 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:47:00.467 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:47:00.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:00.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:01.483 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:47:01.509 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1850,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:47:01.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:01.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:02.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:02.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:03.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:03.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:04.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:47:04.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:47:04.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:04.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:05.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:05.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:07.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:07.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:07.609 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1854,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:47:07.640 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1854,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:47:07.640 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:47:07.640 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:47:07.641 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:47:08.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:08.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:08.605 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1854,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:47:08.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:47:08.627 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:47:09.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:09.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:09.569 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1854,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:47:09.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:47:10.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:10.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:10.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:10.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:11.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:11.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:12.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:12.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:13.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:13.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:14.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:14.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:15.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:15.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:16.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:16.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:17.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:17.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:18.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:18.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:19.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:19.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:20.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:20.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:21.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:21.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:22.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:22.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:23.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:23.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:24.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:24.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:25.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:25.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:26.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:26.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:27.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:27.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:28.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:28.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:29.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:29.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:30.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:30.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:32.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:32.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:33.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:33.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:34.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:34.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:35.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:35.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:36.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:36.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:37.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:37.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:38.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:38.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:39.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:39.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:40.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:40.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:41.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:41.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:42.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:42.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:43.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:43.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:44.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:44.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:45.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:45.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:46.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:46.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:47.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:47.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:48.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:48.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:49.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:49.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:50.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:50.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:51.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:51.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:52.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:52.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:53.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:53.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:54.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:54.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:55.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:55.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:56.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:56.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:57.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:57.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:58.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:58.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:47:59.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:47:59.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:00.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:00.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:01.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:01.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:02.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:02.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:03.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:03.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:04.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:04.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:05.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:05.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:06.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:06.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:07.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:07.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:08.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:08.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:09.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:09.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:10.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:10.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:11.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:11.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:12.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:12.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:13.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:13.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:14.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:14.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:15.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:15.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:16.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:16.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:17.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:17.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:18.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:18.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:19.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:19.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:20.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:20.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:21.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:21.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:22.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:22.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:23.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:23.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:24.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:24.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:25.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:25.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:26.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:26.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:27.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:27.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:28.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:28.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:29.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:29.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:30.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:30.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:31.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:31.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:32.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:32.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:33.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:33.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:34.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:34.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:34.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:34.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:36.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:36.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:37.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:37.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:38.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:38.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:39.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:39.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:39.592 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:48:39.593 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:48:39.593 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:48:39.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:39.593 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:48:39.599 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:48:39.599 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:48:39.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:39.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:48:40.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:40.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:40.577 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1854,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:40.599 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1854,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:41.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:41.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:42.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:42.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:43.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:43.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:43.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:43.601 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:48:44.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:44.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:45.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:45.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:46.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:46.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:47.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:47.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:48.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:48.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:48.510 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1858,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:48.531 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1858,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:49.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:49.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:50.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:50.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:51.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:51.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:51.532 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:51.532 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:48:52.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:52.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:52.521 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:52.543 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:52.544 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:48:52.544 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:52.544 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:48:53.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:53.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:53.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1860,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:48:53.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:53.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:48:54.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:54.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:54.531 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1860,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:48:54.552 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:54.552 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:48:55.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:55.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:55.536 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1860,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:48:55.558 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:48:55.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:55.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:48:56.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:56.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:56.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:48:57.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:57.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:57.670 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:48:57.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:57.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:48:58.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:58.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:58.686 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:58.716 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:48:58.716 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:48:58.717 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:48:58.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:58.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:48:59.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:48:59.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:48:59.723 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1860,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:48:59.754 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:48:59.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:49:00.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:00.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:00.750 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1860,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:49:00.773 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:00.773 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:49:01.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:01.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:01.729 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1860,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:49:01.753 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:49:01.753 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:49:01.753 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:49:01.753 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:49:01.753 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:49:01.753 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:49:01.775 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:49:01.775 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:49:01.776 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:01.776 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:49:02.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:02.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:02.847 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:02.870 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:03.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:03.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:04.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:04.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:05.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:05.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:05.872 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:05.872 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:49:06.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:06.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:06.879 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:06.909 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:07.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:07.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:08.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:08.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:09.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:09.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:09.911 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:09.911 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:49:10.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:10.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:10.937 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:10.960 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1860,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:11.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:11.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:12.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:12.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:13.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:13.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:13.961 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:13.961 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:49:14.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:14.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:15.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:15.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:16.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:16.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:17.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:17.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:17.409 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:17.432 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:49:17.432 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:49:17.432 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:17.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:49:18.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:18.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:18.417 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1864,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:49:18.449 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:49:18.449 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:49:19.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:19.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:19.419 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1864,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:49:19.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:49:20.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:20.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:21.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:21.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:22.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:22.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:23.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:23.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:24.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:24.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:25.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:25.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:26.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:26.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:27.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:27.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:28.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:28.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:29.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:29.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:30.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:30.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:31.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:31.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:32.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:32.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:33.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:33.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:34.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:34.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:35.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:35.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:36.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:36.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:37.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:37.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:38.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:38.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:39.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:39.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:40.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:40.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:41.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:41.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:42.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:42.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:43.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:43.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:44.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:44.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:45.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:45.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:46.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:46.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:47.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:47.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:48.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:48.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:49.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:49.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:50.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:50.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:51.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:51.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:52.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:52.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:53.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:53.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:54.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:54.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:55.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:55.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:56.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:56.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:57.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:57.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:58.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:58.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:49:59.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:49:59.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:00.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:00.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:01.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:01.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:02.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:02.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:03.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:03.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:04.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:04.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:05.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:05.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:06.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:06.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:07.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:07.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:08.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:08.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:09.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:09.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:10.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:10.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:11.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:11.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:12.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:12.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:13.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:13.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:14.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:14.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:15.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:15.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:16.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:16.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:17.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:17.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:18.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:18.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:19.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:19.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:20.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:20.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:21.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:21.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:22.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:22.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:23.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:23.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:24.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:24.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:25.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:25.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:26.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:26.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:27.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:27.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:28.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:28.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:29.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:29.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:30.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:30.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:31.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:31.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:32.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:32.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:33.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:33.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:34.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:34.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:35.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:35.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:36.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:36.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:37.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:37.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:38.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:38.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:39.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:39.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:40.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:40.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:41.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:41.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:42.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:42.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:43.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:43.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:44.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:44.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:45.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:45.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:46.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:46.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:47.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:47.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:48.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:48.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:49.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:49.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:49.442 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:50:49.442 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:50:49.443 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:50:49.443 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:49.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:50:49.450 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:50:49.450 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:50:49.451 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:49.451 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:50:50.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:50.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:50.410 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:50.443 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:51.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:51.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:52.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:52.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:53.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:53.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:53.444 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:53.444 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:50:54.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:54.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:54.433 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:54.458 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1864,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:55.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:55.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:56.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:56.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:57.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:57.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:57.459 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:57.459 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:50:58.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:58.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:58.429 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:58.452 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:50:58.453 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:50:58.453 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:58.453 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:50:59.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:50:59.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:50:59.484 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1866,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:50:59.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:50:59.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:51:00.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:00.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:00.526 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1866,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:00.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:00.559 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:51:01.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:01.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:01.577 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1866,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:01.599 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:51:01.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:01.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:51:02.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:02.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:02.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:51:03.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:03.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:03.719 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:51:03.719 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:03.719 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:51:04.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:04.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:04.715 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:04.737 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:04.737 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:51:04.737 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:51:04.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:04.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:51:05.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:05.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:05.728 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1866,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:05.752 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:05.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:51:06.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:06.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:06.739 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1866,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:06.772 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:06.772 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:51:07.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:07.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:07.726 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1866,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:07.748 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:51:07.748 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:51:07.748 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:51:07.748 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:51:07.749 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:51:07.749 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:51:07.764 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:51:07.764 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:51:07.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:07.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:51:08.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:08.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:08.823 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:08.845 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:09.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:09.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:10.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:10.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:11.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:11.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:11.846 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:11.850 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:51:12.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:12.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:12.816 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:12.839 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1866,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:13.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:13.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:14.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:14.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:15.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:15.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:15.840 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:15.841 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:51:16.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:16.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:17.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:17.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:18.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:18.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:19.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:19.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:20.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:20.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:21.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:21.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:21.838 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1870,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:21.871 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1870,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:51:21.871 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:51:21.871 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:21.871 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:51:22.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:22.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:22.861 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1870,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:22.887 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:51:22.888 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:51:23.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:23.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:23.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1870,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:51:23.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:51:24.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:24.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:25.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:25.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:26.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:26.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:27.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:27.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:28.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:28.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:29.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:29.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:30.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:30.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:31.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:31.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:32.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:32.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:33.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:33.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:34.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:34.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:35.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:35.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:36.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:36.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:37.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:37.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:38.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:38.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:39.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:39.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:40.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:40.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:41.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:41.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:42.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:42.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:43.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:43.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:44.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:44.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:45.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:45.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:46.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:46.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:47.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:47.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:48.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:48.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:49.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:49.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:50.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:50.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:51.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:51.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:52.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:52.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:53.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:53.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:54.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:54.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:55.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:55.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:56.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:56.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:57.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:57.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:58.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:58.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:51:59.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:51:59.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:00.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:00.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:01.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:01.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:02.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:02.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:03.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:03.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:04.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:04.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:05.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:05.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:06.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:06.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:07.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:07.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:08.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:08.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:09.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:09.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:10.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:10.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:11.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:11.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:12.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:12.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:13.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:13.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:14.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:14.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:15.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:15.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:16.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:16.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:17.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:17.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:18.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:18.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:19.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:19.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:20.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:20.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:21.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:21.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:22.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:22.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:23.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:23.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:24.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:24.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:25.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:25.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:26.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:26.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:27.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:27.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:28.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:28.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:29.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:29.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:30.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:30.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:31.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:31.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:32.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:32.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:33.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:33.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:34.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:34.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:35.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:35.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:36.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:36.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:37.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:37.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:38.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:38.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:39.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:39.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:40.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:40.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:41.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:41.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:42.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:42.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:43.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:43.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:44.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:44.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:45.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:45.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:46.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:46.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:47.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:47.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:48.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:48.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:49.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:49.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:50.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:50.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:51.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:51.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:52.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:52.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:53.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:53.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:53.905 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:52:53.906 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:52:53.906 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:52:53.906 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:52:53.906 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:52:53.915 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:52:53.915 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:52:53.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:52:53.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:52:54.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:54.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:55.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:55.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:56.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:56.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:57.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:57.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:58.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:58.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:52:59.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:52:59.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:00.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:00.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:01.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:01.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:02.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:02.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:03.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:03.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:04.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:04.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:05.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:05.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:05.385 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:05.407 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:05.407 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:53:05.408 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:05.408 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:53:06.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:06.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:06.314 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1882,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:06.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:06.345 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:53:07.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:07.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:07.336 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1882,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:07.367 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:07.367 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:53:08.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:08.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:08.362 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1882,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:08.391 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:53:08.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:08.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:53:09.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:09.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:09.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:53:10.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:10.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:10.514 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:53:10.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:10.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:53:11.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:11.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:11.508 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:11.530 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:11.530 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:53:11.530 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:53:11.530 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:11.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:53:12.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:12.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:12.520 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1882,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:12.544 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:12.545 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:53:13.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:13.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:13.530 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1882,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:13.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:13.562 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:53:14.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:14.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:14.542 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1882,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:14.564 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T09:53:14.565 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:53:14.565 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T09:53:14.565 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:53:14.565 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T09:53:14.565 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:53:14.582 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T09:53:14.583 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:53:14.583 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:14.583 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:53:15.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:15.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:15.617 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:15.639 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:16.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:16.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:17.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:17.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:18.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:18.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:18.640 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:18.640 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:53:19.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:19.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:19.634 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:19.656 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:20.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:20.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:21.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:21.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:22.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:22.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:22.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:22.658 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:53:23.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:23.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:23.661 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:23.684 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1882,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:24.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:24.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:25.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:25.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:26.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:26.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:26.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:26.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:53:27.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:27.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:28.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:28.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:29.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:29.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:29.687 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1886,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:29.710 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1886,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:53:29.710 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:53:29.710 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:29.711 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:53:30.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:30.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:30.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1886,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:30.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:53:30.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:53:31.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:31.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:31.707 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1886,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:53:31.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:53:32.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:32.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:33.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:33.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:34.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:34.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:35.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:35.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:36.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:36.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:37.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:37.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:38.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:38.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:39.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:39.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:40.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:40.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:41.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:41.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:42.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:42.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:43.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:43.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:44.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:44.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:45.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:45.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:46.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:46.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:47.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:47.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:48.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:48.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:49.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:49.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:50.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:50.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:51.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:51.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:52.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:52.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:53.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:53.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:54.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:54.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:55.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:55.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:56.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:56.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:57.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:57.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:58.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:58.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:53:59.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:53:59.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:00.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:00.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:01.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:01.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:02.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:02.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:03.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:03.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:04.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:04.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:05.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:05.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:06.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:06.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:07.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:07.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:08.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:08.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:09.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:09.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:10.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:10.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:11.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:11.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:12.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:12.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:13.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:13.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:14.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:14.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:15.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:15.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:16.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:16.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:17.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:17.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:18.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:18.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:19.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:19.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:20.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:20.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:21.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:21.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:22.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:22.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:23.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:23.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:24.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:24.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:25.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:25.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:26.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:26.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:27.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:27.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:28.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:28.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:29.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:29.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:30.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:30.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:31.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:31.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:32.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:32.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:33.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:33.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:34.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:34.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:35.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:35.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:36.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:36.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:37.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:37.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:38.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:38.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:39.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:39.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:40.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:40.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:41.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:41.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:42.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:42.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:43.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:43.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:44.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:44.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:45.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:45.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:46.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:46.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:47.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:47.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:48.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:48.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:49.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:49.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:50.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:50.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:51.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:51.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:52.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:52.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:53.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:53.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:54.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:54.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:55.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:55.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:56.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:56.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:57.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:57.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:58.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:58.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:54:59.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:54:59.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:00.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:00.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:01.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:01.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:01.731 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T09:55:01.731 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T09:55:01.731 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T09:55:01.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:01.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T09:55:01.738 INFO:tasks.ceph.mon.c:Started 2019-10-01T09:55:01.738 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:55:01.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:01.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:02.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:02.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:02.701 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1886,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:02.732 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1886,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:03.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:03.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:04.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:04.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:05.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:05.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:05.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:05.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:06.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:06.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:07.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:07.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:08.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:08.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:09.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:09.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:10.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:10.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:10.710 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:10.732 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:10.732 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:55:10.733 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:10.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:55:11.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:11.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:11.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1892,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:11.754 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:11.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:55:12.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:12.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:12.916 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1892,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:12.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:12.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:55:13.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:13.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:13.926 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1892,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:13.957 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:55:13.957 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:13.957 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:55:14.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:14.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:15.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:55:15.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:15.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:16.059 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:55:16.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:16.060 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:16.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:16.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:17.066 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:17.095 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:17.095 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:55:17.095 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:55:17.095 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:17.095 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:55:17.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:17.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:18.075 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1892,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:18.105 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:18.105 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:55:18.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:18.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:19.093 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1892,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:19.126 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:19.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:55:19.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:19.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:20.142 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1892,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:20.172 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T09:55:20.172 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:55:20.173 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T09:55:20.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:55:20.173 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T09:55:20.173 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:55:20.198 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T09:55:20.198 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:55:20.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:20.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:20.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:20.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:21.280 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:21.302 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:21.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:21.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:22.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:22.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:23.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:23.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:24.304 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:24.305 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:24.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:24.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:25.291 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:25.313 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:25.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:25.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:26.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:26.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:27.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:27.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:28.314 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:28.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:28.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:28.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:29.318 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:29.341 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1892,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:29.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:29.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:30.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:30.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:31.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:31.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:32.342 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:32.342 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:55:32.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:32.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:33.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:33.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:34.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:34.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:35.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:35.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:36.387 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:36.418 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:55:36.418 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:55:36.419 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:36.419 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:55:36.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:36.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:37.413 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1896,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:37.440 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:55:37.440 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:55:37.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:37.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:38.425 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1896,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:55:38.456 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:55:38.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:38.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:39.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:39.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:40.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:40.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:41.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:41.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:42.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:42.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:43.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:43.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:44.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:44.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:45.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:45.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:46.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:46.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:47.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:47.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:48.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:48.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:49.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:49.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:50.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:50.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:51.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:51.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:52.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:52.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:53.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:53.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:54.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:54.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:55.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:55.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:56.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:56.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:57.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:57.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:58.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:58.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:55:59.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:55:59.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:00.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:00.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:01.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:01.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:02.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:02.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:03.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:03.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:04.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:04.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:05.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:05.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:06.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:06.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:07.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:07.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:08.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:08.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:09.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:09.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:10.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:10.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:11.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:11.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:12.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:12.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:13.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:13.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:14.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:14.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:15.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:15.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:16.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:16.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:17.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:17.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:18.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:18.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:19.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:19.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:20.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:20.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:21.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:21.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:22.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:22.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:23.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:23.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:24.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:24.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:25.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:25.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:26.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:26.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:27.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:27.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:28.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:28.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:29.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:29.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:30.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:30.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:31.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:31.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:32.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:32.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:33.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:33.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:34.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:34.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:35.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:35.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:36.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:36.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:37.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:37.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:38.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:38.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:39.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:39.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:40.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:40.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:41.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:41.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:42.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:42.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:43.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:43.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:44.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:44.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:45.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:45.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:46.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:46.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:47.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:47.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:48.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:48.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:49.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:49.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:50.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:50.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:51.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:51.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:52.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:52.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:53.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:53.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:54.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:54.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:55.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:55.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:56.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:56.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:57.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:57.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:58.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:58.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:56:59.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:56:59.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:00.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:00.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:01.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:01.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:02.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:02.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:03.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:03.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:04.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:04.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:05.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:05.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:06.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:06.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:07.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:07.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:08.456 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T09:57:08.457 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T09:57:08.457 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T09:57:08.457 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:08.457 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T09:57:08.466 INFO:tasks.ceph.mon.a:Started 2019-10-01T09:57:08.466 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:57:08.466 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:08.466 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:08.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:08.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:09.488 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:09.509 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:09.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:09.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:10.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:10.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:11.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:11.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:12.510 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:12.511 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:12.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:12.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:13.526 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:13.548 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1896,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:13.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:13.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:14.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:14.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:15.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:15.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:16.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:16.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:16.549 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:16.549 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:17.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:17.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:17.622 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:17.654 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:17.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:57:17.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:17.654 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:57:18.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:18.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:18.774 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1898,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:18.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:18.805 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:57:19.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:19.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:19.807 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1898,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:19.838 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:19.838 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:57:20.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:20.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:20.848 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1898,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:20.878 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:57:20.878 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:20.878 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:57:21.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:21.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:21.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:57:22.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:22.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:22.981 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:57:22.981 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:22.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:23.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:23.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:23.973 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:24.004 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:24.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:57:24.004 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:57:24.005 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:24.005 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:57:24.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:24.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:24.990 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1898,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:25.013 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:25.013 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:57:25.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:25.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:26.008 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1898,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:26.030 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:26.030 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:57:26.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:26.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:27.056 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1898,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:27.078 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:57:27.079 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:57:27.079 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:57:27.079 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:57:27.079 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:57:27.079 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:57:27.116 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:57:27.116 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:57:27.116 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:27.116 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:27.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:27.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:28.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:28.147 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:28.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:28.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:29.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:29.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:30.555 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:30.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:31.148 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:31.148 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:31.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:31.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:32.128 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:32.151 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1898,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:32.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:32.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:33.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:33.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:34.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:34.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:35.152 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:35.152 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:57:35.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:35.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:36.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:36.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:37.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:37.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:38.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:38.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:39.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:39.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:40.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:40.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:41.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:41.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:42.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:42.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:42.718 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:42.748 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:57:42.748 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:57:42.749 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:42.749 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:57:43.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:43.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:43.738 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1900,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:57:43.758 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:57:43.759 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:57:44.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:44.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:44.725 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1900,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:57:44.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:57:45.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:45.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:46.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:46.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:47.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:47.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:48.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:48.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:49.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:49.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:50.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:50.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:51.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:51.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:52.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:52.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:53.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:53.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:54.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:54.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:55.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:55.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:56.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:56.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:57.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:57.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:58.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:58.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:57:59.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:57:59.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:00.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:00.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:01.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:01.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:02.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:02.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:03.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:03.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:04.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:04.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:05.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:05.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:06.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:06.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:07.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:07.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:08.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:08.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:09.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:09.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:10.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:10.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:11.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:11.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:12.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:12.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:13.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:13.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:14.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:14.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:15.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:15.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:16.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:16.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:17.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:17.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:18.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:18.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:19.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:19.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:20.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:20.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:21.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:21.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:22.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:22.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:23.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:23.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:24.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:24.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:25.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:25.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:26.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:26.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:27.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:27.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:28.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:28.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:29.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:29.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:30.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:30.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:31.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:31.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:32.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:32.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:33.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:33.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:34.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:34.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:35.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:35.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:36.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:36.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:38.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:38.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:39.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:39.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:40.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:40.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:41.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:41.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:42.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:42.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:43.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:43.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:44.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:44.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:45.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:45.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:46.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:46.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:47.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:47.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:48.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:48.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:49.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:49.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:50.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:50.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:51.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:51.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:52.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:52.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:53.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:53.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:54.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:54.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:55.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:55.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:56.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:56.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:57.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:57.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:58.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:58.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:58:59.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:58:59.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:00.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:00.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:01.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:01.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:02.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:02.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:03.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:03.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:04.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:04.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:05.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:05.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:06.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:06.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:07.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:07.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:08.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:08.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:08.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:08.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:09.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:09.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:10.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:10.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:11.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:11.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:12.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:12.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:13.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:13.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:14.758 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:59:14.758 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T09:59:14.758 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T09:59:14.759 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T09:59:14.759 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T09:59:14.765 INFO:tasks.ceph.mon.b:Started 2019-10-01T09:59:14.766 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:59:14.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:14.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:14.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:14.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:15.770 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:15.800 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:15.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:15.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:16.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:16.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:17.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:17.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:18.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:18.801 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:18.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:18.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:19.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:19.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:19.800 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:19.831 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:20.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:20.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:21.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:21.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:22.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:22.832 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:22.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:22.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:23.825 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:23.856 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:23.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:23.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:24.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:24.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:25.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:25.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:26.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:26.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:26.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:26.857 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:27.816 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:27.838 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1900,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:27.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:27.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:28.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:28.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:29.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:29.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:30.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:30.840 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:30.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:30.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:31.858 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:31.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:31.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:31.890 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:31.890 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:59:31.891 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:31.891 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:59:32.896 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1904,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:59:32.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:32.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:32.921 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:32.921 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:59:33.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:33.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:33.938 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1904,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:59:33.970 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:33.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:59:34.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:34.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:34.978 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1904,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:59:35.001 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T09:59:35.001 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:35.001 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T09:59:35.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:35.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:36.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T09:59:36.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:36.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:37.077 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T09:59:37.077 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:37.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:37.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:37.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:38.066 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:38.088 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:38.089 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T09:59:38.089 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T09:59:38.089 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:38.090 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:59:39.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:38.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:39.074 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1904,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:59:39.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:39.107 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:59:39.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:39.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:40.094 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1904,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:59:40.120 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:40.120 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T09:59:40.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:40.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:41.097 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1904,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:59:41.119 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T09:59:41.119 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T09:59:41.119 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T09:59:41.120 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T09:59:41.120 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T09:59:41.120 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T09:59:41.139 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T09:59:41.140 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T09:59:41.140 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:41.140 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:42.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:42.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:42.146 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:42.169 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:42.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:42.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:43.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:43.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:45.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:45.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:45.170 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:45.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:46.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:46.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:46.124 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:46.154 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:47.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:47.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:48.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:48.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:49.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:49.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:49.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:49.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:50.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:50.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:50.171 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:50.203 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1904,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:51.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:51.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:52.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:52.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:53.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:53.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:53.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:53.204 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T09:59:54.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:54.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:55.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:55.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:56.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:56.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:56.207 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1906,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:56.239 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1906,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T09:59:56.239 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T09:59:56.239 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:56.240 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T09:59:57.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:56.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:57.259 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1906,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T09:59:57.282 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T09:59:57.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T09:59:58.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:58.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:58.284 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1906,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T09:59:58.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T09:59:58.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:58.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T09:59:59.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T09:59:59.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:00.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:00.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:01.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:01.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:02.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:02.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:03.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:03.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:04.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:04.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:05.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:05.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:06.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:06.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:07.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:07.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:08.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:08.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:09.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:09.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:10.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:10.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:12.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:12.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:12.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:12.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:14.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:14.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:15.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:15.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:16.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:16.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:16.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:16.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:17.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:17.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:18.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:18.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:19.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:19.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:20.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:20.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:21.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:21.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:22.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:22.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:23.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:23.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:24.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:24.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:25.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:25.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:26.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:26.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:27.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:27.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:28.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:28.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:29.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:29.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:30.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:30.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:31.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:31.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:32.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:32.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:33.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:33.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:34.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:34.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:35.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:35.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:36.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:36.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:37.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:37.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:38.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:38.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:39.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:39.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:40.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:40.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:41.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:41.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:42.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:42.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:43.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:43.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:44.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:44.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:45.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:45.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:46.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:46.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:47.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:47.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:48.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:48.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:49.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:49.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:50.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:50.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:51.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:51.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:52.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:52.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:53.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:53.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:54.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:54.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:55.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:55.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:56.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:56.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:57.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:57.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:58.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:58.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:00:59.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:00:59.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:00.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:00.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:01.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:01.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:02.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:02.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:03.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:03.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:04.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:04.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:05.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:05.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:06.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:06.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:07.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:07.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:08.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:08.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:09.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:09.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:10.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:10.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:11.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:11.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:12.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:12.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:13.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:13.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:14.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:14.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:15.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:15.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:16.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:16.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:17.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:17.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:18.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:18.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:19.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:19.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:20.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:20.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:21.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:21.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:22.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:22.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:23.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:23.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:24.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:24.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:25.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:25.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:26.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:26.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:27.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:27.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:28.307 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:01:28.307 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:01:28.307 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:01:28.307 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:01:28.308 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:01:28.316 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:01:28.316 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:01:28.316 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:28.316 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:01:28.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:28.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:29.356 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1906,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:29.379 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1906,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:29.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:29.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:30.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:30.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:31.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:31.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:32.380 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:32.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:01:32.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:32.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:33.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:33.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:34.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:34.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:35.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:35.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:36.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:36.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:37.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:37.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:38.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:38.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:39.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:39.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:40.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:40.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:41.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:41.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:42.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:42.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:43.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:43.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:44.013 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1914,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:44.035 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1914,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:44.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:44.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:45.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:45.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:46.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:46.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:47.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:47.036 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:01:47.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:47.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:48.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:48.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:49.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:49.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:50.046 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1918,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:50.068 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1918,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:50.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:50.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:51.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:51.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:52.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:52.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:53.069 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:53.070 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:01:53.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:53.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:54.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:54.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:54.745 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:54.777 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:01:54.777 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:01:54.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:54.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:01:55.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:55.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:55.732 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1922,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:01:55.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:55.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:01:56.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:56.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:56.734 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1922,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:01:56.756 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:56.756 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:01:57.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:57.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:57.765 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1922,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:01:57.796 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:01:57.796 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:57.796 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:01:58.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:58.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:58.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:01:59.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:01:59.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:01:59.932 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:01:59.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:01:59.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:02:00.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:00.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:01.014 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:01.039 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:01.039 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:02:01.039 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:02:01.039 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:01.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:02:01.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:01.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:02.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1922,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:02:02.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:02.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:02:02.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:02.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:03.114 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1922,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:02:03.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:03.137 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:02:03.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:03.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:04.153 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1922,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:02:04.175 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:02:04.175 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:02:04.175 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:02:04.176 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:02:04.176 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:02:04.176 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:02:04.195 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:02:04.195 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:02:04.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:04.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:02:04.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:04.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:05.176 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:05.198 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:05.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:05.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:06.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:06.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:07.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:07.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:08.199 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:08.199 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:02:08.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:08.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:09.195 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:09.226 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:09.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:09.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:10.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:10.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:11.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:11.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:12.229 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:12.229 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:02:12.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:12.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:13.219 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:13.238 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1922,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:13.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:13.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:14.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:14.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:15.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:15.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:16.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:16.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:02:16.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:16.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:17.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:17.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:18.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:18.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:19.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:19.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:20.300 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1924,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:20.321 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1924,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:02:20.321 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:02:20.322 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:20.322 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:02:20.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:20.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:21.314 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1924,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:02:21.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:02:21.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:02:21.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:21.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:22.349 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1924,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:02:22.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:02:22.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:22.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:23.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:23.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:24.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:24.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:25.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:25.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:26.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:26.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:27.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:27.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:28.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:28.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:29.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:29.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:30.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:30.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:31.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:31.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:32.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:32.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:33.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:33.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:34.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:34.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:35.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:35.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:36.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:36.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:37.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:37.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:38.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:38.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:39.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:39.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:40.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:40.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:41.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:41.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:42.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:42.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:43.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:43.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:44.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:44.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:45.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:45.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:46.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:46.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:47.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:47.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:48.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:48.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:49.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:49.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:50.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:50.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:51.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:51.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:52.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:52.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:53.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:53.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:54.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:54.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:55.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:55.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:56.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:56.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:57.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:57.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:58.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:58.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:02:59.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:02:59.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:00.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:00.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:01.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:01.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:02.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:02.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:03.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:03.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:04.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:04.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:05.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:05.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:06.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:06.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:07.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:07.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:08.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:08.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:09.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:09.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:10.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:10.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:11.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:11.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:12.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:12.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:13.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:13.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:14.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:14.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:15.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:15.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:16.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:16.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:17.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:17.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:18.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:18.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:19.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:19.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:20.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:20.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:21.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:21.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:22.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:22.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:23.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:23.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:24.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:24.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:25.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:25.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:26.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:26.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:27.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:27.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:28.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:28.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:29.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:29.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:30.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:30.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:31.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:31.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:32.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:32.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:33.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:33.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:34.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:34.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:35.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:35.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:36.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:36.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:37.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:37.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:38.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:38.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:39.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:39.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:40.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:40.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:41.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:41.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:42.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:42.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:43.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:43.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:44.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:44.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:45.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:45.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:46.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:46.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:47.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:47.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:48.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:48.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:49.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:49.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:50.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:50.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:51.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:51.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:52.380 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:03:52.380 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:03:52.380 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:03:52.381 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:03:52.381 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:03:52.387 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:03:52.387 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:03:52.388 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:03:52.388 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:03:52.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:52.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:53.378 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1924,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:03:53.402 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1924,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:03:53.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:53.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:54.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:54.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:55.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:55.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:56.403 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:03:56.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:03:56.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:56.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:57.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:57.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:58.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:58.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:03:59.586 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:03:59.608 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:03:59.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:03:59.609 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:03:59.609 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:03:59.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:03:59.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:00.604 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1928,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:00.626 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:00.627 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:04:00.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:00.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:01.625 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1928,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:01.655 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:01.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:04:01.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:01.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:02.649 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1928,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:02.671 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:04:02.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:02.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:04:02.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:02.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:03.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:04:03.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:03.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:04.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:04.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:04.769 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:04:04.769 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:04.769 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:04:05.631 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:05.653 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:05.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:04:05.654 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:04:05.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:05.654 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:04:05.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:05.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:06.580 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1928,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:06.610 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:06.610 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:04:06.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:06.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:07.561 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1928,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:07.584 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:07.584 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:04:07.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:07.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:08.503 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1928,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:08.525 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:04:08.526 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:04:08.526 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:04:08.526 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:04:08.526 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:04:08.526 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:04:08.544 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:04:08.544 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:04:08.545 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:08.545 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:04:08.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:08.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:09.537 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:09.559 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:09.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:09.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:10.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:10.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:11.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:11.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:12.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:12.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:04:12.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:12.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:13.569 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:13.591 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:13.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:13.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:14.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:14.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:15.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:15.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:16.596 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:16.596 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:04:16.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:16.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:17.600 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:17.630 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1928,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:17.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:17.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:18.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:18.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:19.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:19.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:20.631 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:20.631 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:04:20.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:20.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:21.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:21.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:22.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:22.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:23.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:23.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:24.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:24.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:24.703 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:24.728 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:04:24.728 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:04:24.729 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:24.729 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:04:25.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:25.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:25.800 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1930,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:25.822 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:04:25.822 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:04:26.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:26.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:26.831 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1930,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:04:26.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:04:27.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:27.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:28.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:28.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:29.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:29.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:30.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:30.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:31.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:31.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:32.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:32.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:33.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:33.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:34.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:34.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:35.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:35.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:36.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:36.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:37.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:37.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:38.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:38.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:39.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:39.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:40.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:40.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:41.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:41.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:42.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:42.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:43.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:43.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:44.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:44.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:45.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:45.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:46.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:46.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:47.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:47.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:48.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:48.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:49.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:49.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:50.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:50.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:51.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:51.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:52.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:52.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:53.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:53.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:54.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:54.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:55.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:55.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:56.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:56.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:57.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:57.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:58.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:58.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:04:59.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:04:59.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:00.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:00.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:01.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:01.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:02.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:02.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:03.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:03.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:04.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:04.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:05.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:05.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:06.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:06.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:07.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:07.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:08.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:08.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:09.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:09.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:10.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:10.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:11.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:11.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:12.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:12.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:13.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:13.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:14.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:14.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:15.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:15.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:16.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:16.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:17.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:17.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:18.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:18.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:19.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:19.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:20.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:20.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:21.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:21.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:22.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:22.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:23.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:23.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:24.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:24.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:25.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:25.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:26.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:26.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:27.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:27.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:28.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:28.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:29.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:29.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:30.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:30.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:31.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:31.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:32.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:32.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:33.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:33.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:34.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:34.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:35.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:35.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:36.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:36.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:37.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:37.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:38.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:38.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:39.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:39.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:40.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:40.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:41.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:41.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:42.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:42.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:43.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:43.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:44.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:44.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:45.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:45.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:46.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:46.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:47.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:47.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:48.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:48.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:50.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:50.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:51.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:51.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:52.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:52.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:52.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:52.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:53.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:53.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:54.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:54.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:55.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:55.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:56.854 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:05:56.854 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:05:56.855 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:05:56.855 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:05:56.855 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:05:56.861 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:05:56.861 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:05:56.861 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:05:56.861 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:05:56.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:56.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:57.868 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:05:57.900 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:05:57.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:57.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:58.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:58.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:05:59.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:05:59.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:00.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:00.901 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:06:01.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:01.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:01.893 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:01.923 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1930,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:01.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:01.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:03.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:03.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:04.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:04.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:04.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:04.925 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:06:05.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:05.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:05.915 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:05.945 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:05.945 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:06:05.946 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:05.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:06:06.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:06.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:06.941 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1934,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:06.959 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:06.959 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:06:06.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:06.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:07.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:07.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:07.962 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1934,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:07.984 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:07.984 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:06:08.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:08.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:09.060 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1934,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:09.090 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:06:09.090 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:09.090 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:06:09.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:09.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:10.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:06:10.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:10.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:11.190 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:06:11.190 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:11.190 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:06:11.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:11.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:12.173 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:12.198 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1934,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:12.198 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:06:12.198 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:06:12.198 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:12.198 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:06:12.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:12.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:13.158 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1934,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:13.182 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:13.182 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:06:13.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:13.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:14.184 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1934,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:14.203 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:14.203 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:06:14.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:14.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:15.230 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1934,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:15.260 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:06:15.260 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T10:06:15.261 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:06:15.261 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:06:15.261 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:06:15.261 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:06:15.278 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:06:15.278 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:06:15.279 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:06:15.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:06:15.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:15.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:16.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:16.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:17.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:17.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:18.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:18.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:19.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:19.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:20.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:20.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:21.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:21.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:22.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:22.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:23.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:23.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:24.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:24.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:25.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:25.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:26.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:26.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:27.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:27.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:28.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:28.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:29.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:29.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:30.280 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:06:30.280 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:06:30.280 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:06:30.281 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:30.281 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:06:30.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:30.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:31.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:31.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:32.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:32.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:33.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:33.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:34.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:34.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:35.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:35.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:36.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:36.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:37.473 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1936,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:37.503 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1936,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:06:37.503 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:06:37.503 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:37.504 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:06:37.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:37.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:38.514 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1936,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:38.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:06:38.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:06:38.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:38.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:39.553 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1936,"quorum":[1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:06:39.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:06:39.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:39.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:40.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:40.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:42.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:42.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:42.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:42.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:43.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:43.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:44.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:44.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:45.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:45.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:46.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:46.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:47.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:47.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:48.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:48.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:49.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:49.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:51.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:51.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:51.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:51.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:52.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:52.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:53.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:53.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:54.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:54.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:55.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:55.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:56.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:56.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:57.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:57.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:58.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:58.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:06:59.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:06:59.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:00.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:00.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:01.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:01.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:02.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:02.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:03.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:03.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:04.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:04.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:05.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:05.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:06.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:06.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:07.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:07.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:08.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:08.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:09.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:09.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:10.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:10.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:11.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:11.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:12.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:12.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:13.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:13.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:14.789 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:14.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:15.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:15.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:16.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:16.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:17.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:17.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:18.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:18.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:19.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:19.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:20.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:20.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:21.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:21.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:22.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:22.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:23.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:23.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:24.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:24.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:25.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:25.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:26.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:26.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:27.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:27.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:28.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:28.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:29.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:29.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:30.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:30.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:31.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:31.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:32.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:32.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:33.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:33.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:34.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:34.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:35.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:35.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:36.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:36.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:37.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:37.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:38.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:38.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:39.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:39.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:40.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:40.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:41.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:41.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:42.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:42.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:43.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:43.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:44.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:44.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:45.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:45.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:46.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:46.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:47.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:47.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:48.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:48.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:49.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:49.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:50.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:50.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:51.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:51.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:52.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:52.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:53.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:53.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:54.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:54.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:55.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:55.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:56.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:56.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:57.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:57.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:58.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:58.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:07:59.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:07:59.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:00.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:00.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:01.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:01.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:02.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:02.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:03.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:03.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:04.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:04.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:05.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:05.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:06.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:06.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:07.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:07.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:08.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:08.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:09.575 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:08:09.576 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:08:09.576 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:08:09.576 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:08:09.576 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:08:09.582 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:08:09.582 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:08:09.583 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:08:09.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:08:09.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:09.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:10.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:10.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:11.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:11.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:12.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:12.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:13.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:13.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:14.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:14.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:15.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:15.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:16.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:16.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:17.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:17.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:18.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:18.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:19.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:19.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:20.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:20.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:21.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:21.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:22.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:22.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:23.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:23.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:24.583 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:08:24.584 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:08:24.584 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:08:24.585 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:24.585 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:24.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:24.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:25.620 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:25.643 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:25.643 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:08:25.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:25.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:08:25.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:25.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:26.665 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1944,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:26.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:26.688 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:08:26.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:26.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:27.688 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1944,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:27.718 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:27.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:08:27.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:27.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:28.728 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1944,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:28.745 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:08:28.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:28.745 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:08:28.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:28.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:29.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:08:29.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:29.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:30.851 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:08:30.852 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:30.852 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:30.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:30.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:31.837 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:31.868 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:31.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:08:31.869 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:08:31.869 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:31.869 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:08:31.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:31.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:32.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1944,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:32.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:32.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:08:32.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:32.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:33.921 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1944,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:33.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:33.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:33.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:33.944 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:08:34.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:34.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:34.966 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1944,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:34.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:08:34.989 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:08:34.989 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:08:34.989 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:08:34.989 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:08:34.990 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:08:35.019 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:08:35.019 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:08:35.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:35.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:35.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:35.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:36.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:36.104 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:36.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:36.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:37.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:37.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:38.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:38.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:39.108 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:39.108 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:39.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:39.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:40.085 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:40.108 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:40.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:40.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:41.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:41.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:42.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:42.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:43.109 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:43.109 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:43.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:43.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:44.101 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:44.129 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1944,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:44.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:44.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:45.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:45.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:46.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:46.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:47.131 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:47.131 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:08:47.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:47.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:48.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:48.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:50.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:50.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:50.569 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1948,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:50.598 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1948,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:08:50.598 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:08:50.599 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:50.599 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:08:51.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:51.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:51.603 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1948,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:51.626 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:08:51.626 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:08:52.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:52.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:52.616 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1948,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:08:52.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:08:53.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:53.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:54.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:54.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:54.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:54.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:56.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:55.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:56.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:56.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:58.014 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:58.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:58.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:58.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:08:59.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:08:59.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:00.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:00.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:01.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:01.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:02.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:02.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:03.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:03.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:04.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:04.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:05.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:05.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:06.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:06.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:07.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:07.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:08.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:08.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:09.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:09.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:10.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:10.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:11.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:11.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:12.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:12.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:13.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:13.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:14.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:14.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:15.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:15.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:16.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:16.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:17.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:17.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:18.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:18.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:19.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:19.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:20.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:20.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:21.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:21.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:22.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:22.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:23.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:23.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:24.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:24.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:25.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:25.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:26.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:26.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:27.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:27.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:28.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:28.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:29.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:29.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:30.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:30.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:32.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:32.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:32.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:32.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:34.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:34.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:35.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:35.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:36.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:36.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:37.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:37.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:38.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:38.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:39.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:39.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:39.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:39.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:41.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:41.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:42.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:42.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:43.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:43.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:44.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:44.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:45.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:45.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:46.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:46.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:47.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:47.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:48.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:48.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:49.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:49.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:50.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:50.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:51.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:51.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:52.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:52.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:53.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:53.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:54.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:54.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:55.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:55.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:56.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:56.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:57.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:57.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:58.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:58.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:09:59.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:09:59.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:00.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:00.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:00.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:00.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:01.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:01.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:02.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:02.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:03.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:03.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:04.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:04.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:05.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:05.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:06.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:06.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:07.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:07.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:08.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:08.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:09.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:09.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:10.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:10.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:11.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:11.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:12.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:12.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:13.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:13.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:14.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:14.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:15.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:15.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:16.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:16.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:17.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:17.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:18.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:18.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:19.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:19.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:20.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:20.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:21.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:21.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:22.639 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:10:22.639 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:10:22.639 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:10:22.639 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:22.640 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:10:22.648 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:10:22.648 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:10:22.648 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:22.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:10:22.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:22.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:23.648 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1948,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:23.675 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1948,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:23.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:23.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:25.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:25.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:26.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:26.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:26.677 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:26.677 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:10:27.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:27.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:28.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:28.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:29.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:29.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:29.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:29.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:30.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:30.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:31.688 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1954,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:31.720 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1954,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:31.720 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:10:31.720 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:31.721 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:10:31.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:31.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:32.731 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1954,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:32.762 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:32.762 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:10:33.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:32.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:33.757 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1954,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:33.786 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:33.786 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:10:34.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:34.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:34.790 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1954,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:34.820 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:10:34.820 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:34.820 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:10:35.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:35.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:35.898 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:10:36.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:36.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:36.899 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:10:36.900 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:36.900 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:10:37.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:37.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:37.901 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1954,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:37.922 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1954,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:10:37.922 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:10:37.922 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:10:37.922 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:37.923 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:10:37.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:37.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:38.932 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1954,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:38.955 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:38.955 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:10:39.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:39.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:39.905 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1954,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:39.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:39.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:10:40.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:40.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:40.910 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1954,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:10:40.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:10:40.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T10:10:40.933 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:10:40.934 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:10:40.934 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:10:40.934 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:10:40.972 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:10:40.972 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:10:40.973 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:10:40.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:10:40.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:40.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:41.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:41.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:42.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:42.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:43.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:43.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:44.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:44.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:45.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:45.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:46.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:46.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:47.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:47.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:48.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:48.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:49.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:49.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:50.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:50.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:51.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:51.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:52.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:52.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:53.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:53.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:54.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:54.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:55.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:55.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:55.973 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:10:55.974 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:10:55.974 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:10:55.974 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:10:55.974 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:10:56.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:56.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:57.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:57.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:58.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:58.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:10:59.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:10:59.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:00.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:00.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:01.645 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1956,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:11:01.666 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1956,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:11:01.666 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:11:01.667 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:11:01.667 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:11:01.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:01.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:02.666 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1956,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:11:02.690 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:11:02.690 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:11:02.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:02.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:03.675 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1956,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:11:03.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:11:03.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:03.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:04.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:04.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:05.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:05.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:06.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:06.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:07.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:07.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:08.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:08.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:09.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:09.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:10.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:10.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:11.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:11.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:12.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:12.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:13.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:13.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:14.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:14.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:16.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:15.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:16.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:16.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:17.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:17.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:18.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:18.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:19.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:19.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:20.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:20.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:21.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:21.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:22.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:22.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:23.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:23.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:24.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:24.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:25.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:25.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:26.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:26.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:27.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:27.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:28.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:28.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:29.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:29.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:30.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:30.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:31.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:31.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:32.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:32.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:33.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:33.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:34.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:34.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:35.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:35.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:36.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:36.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:37.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:37.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:38.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:38.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:39.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:39.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:40.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:40.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:41.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:41.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:42.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:42.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:43.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:43.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:44.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:44.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:45.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:45.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:46.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:46.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:47.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:47.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:48.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:48.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:49.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:49.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:50.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:50.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:51.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:51.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:52.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:52.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:53.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:53.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:54.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:54.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:55.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:55.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:56.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:56.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:57.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:57.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:58.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:58.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:11:59.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:11:59.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:00.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:00.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:01.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:01.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:02.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:02.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:03.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:03.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:04.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:04.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:05.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:05.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:06.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:06.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:07.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:07.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:08.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:08.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:09.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:09.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:10.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:10.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:11.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:11.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:12.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:12.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:13.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:13.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:14.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:14.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:15.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:15.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:16.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:16.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:17.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:17.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:18.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:18.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:19.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:19.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:20.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:20.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:21.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:21.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:23.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:23.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:24.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:24.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:25.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:25.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:26.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:26.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:27.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:27.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:28.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:28.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:29.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:29.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:29.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:29.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:30.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:30.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:32.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:31.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:33.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:33.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:33.706 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:12:33.706 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:12:33.707 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:12:33.707 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:33.707 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:12:33.714 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:12:33.714 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:12:33.714 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:12:33.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:12:34.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:34.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:35.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:35.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:36.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:36.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:37.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:37.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:38.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:38.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:39.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:39.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:40.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:40.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:41.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:41.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:42.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:42.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:43.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:43.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:44.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:44.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:45.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:45.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:46.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:46.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:47.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:47.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:48.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:48.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:48.716 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:12:48.716 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:12:48.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:12:48.716 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:48.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:12:49.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:49.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:49.724 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:12:49.754 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:12:49.754 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:12:49.755 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:49.755 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:12:50.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:50.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:50.759 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1960,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:12:50.787 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:50.787 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:12:51.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:51.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:51.771 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1960,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:12:51.792 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:51.792 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:12:52.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:52.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:52.792 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1960,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:12:52.822 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:12:52.822 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:52.822 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:12:53.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:53.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:53.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:12:54.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:54.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:54.936 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:12:54.937 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:54.937 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:12:55.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:55.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:55.959 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:12:55.980 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:12:55.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:12:55.980 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:12:55.980 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:55.981 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:12:56.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:56.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:56.986 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1960,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:12:57.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:57.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:12:57.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:57.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:58.022 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1960,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:12:58.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:58.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:12:58.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:58.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:12:58.995 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1960,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:12:59.018 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:12:59.018 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:12:59.018 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:12:59.018 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:12:59.019 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:12:59.019 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:12:59.072 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:12:59.072 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:12:59.073 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:12:59.073 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:12:59.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:12:59.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:00.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:00.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:00.101 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:00.122 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:01.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:01.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:02.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:02.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:03.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:03.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:03.124 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:13:03.124 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:13:04.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:04.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:04.123 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:04.154 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1960,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:05.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:05.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:06.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:06.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:07.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:07.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:07.157 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:13:07.157 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:13:08.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:08.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:09.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:09.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:10.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:10.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:11.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:11.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:12.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:12.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:13.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:13.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:13.113 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1964,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:13.146 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1964,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:13:13.146 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:13:13.146 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:13:13.147 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:13:14.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:14.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:14.132 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1964,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:13:14.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:13:14.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:13:15.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:15.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:15.135 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1964,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:13:15.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:13:16.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:16.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:17.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:17.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:18.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:18.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:19.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:19.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:20.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:20.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:21.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:21.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:22.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:22.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:23.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:23.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:24.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:24.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:25.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:25.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:26.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:26.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:27.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:27.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:28.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:28.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:29.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:29.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:30.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:30.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:31.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:31.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:32.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:32.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:33.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:33.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:34.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:34.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:35.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:35.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:36.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:36.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:37.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:37.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:38.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:38.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:39.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:39.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:40.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:40.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:41.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:41.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:42.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:42.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:43.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:43.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:44.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:44.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:45.023 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:45.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:46.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:46.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:47.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:47.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:48.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:48.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:49.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:49.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:50.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:50.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:51.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:51.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:52.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:52.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:53.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:53.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:53.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:53.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:54.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:54.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:56.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:56.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:56.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:56.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:57.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:57.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:58.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:58.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:13:59.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:13:59.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:00.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:00.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:01.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:01.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:02.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:02.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:03.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:03.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:04.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:04.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:05.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:05.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:06.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:06.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:07.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:07.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:08.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:08.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:09.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:09.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:10.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:10.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:11.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:11.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:12.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:12.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:13.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:13.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:14.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:14.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:15.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:15.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:16.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:16.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:17.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:17.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:18.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:18.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:19.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:19.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:20.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:20.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:21.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:21.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:22.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:22.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:23.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:23.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:24.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:24.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:25.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:25.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:26.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:26.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:27.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:27.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:28.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:28.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:29.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:29.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:30.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:30.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:31.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:31.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:32.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:32.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:33.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:33.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:34.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:34.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:35.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:35.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:36.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:36.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:37.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:37.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:38.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:38.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:39.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:39.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:40.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:40.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:41.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:41.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:42.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:42.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:43.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:43.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:44.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:44.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:45.167 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:14:45.168 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:14:45.168 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:14:45.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:45.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:14:45.174 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:14:45.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:14:45.175 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:45.175 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:14:45.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:45.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:46.166 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1964,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:46.189 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1964,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:46.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:46.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:47.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:47.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:48.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:48.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:49.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:49.190 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:14:49.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:49.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:50.199 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:50.224 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:50.224 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:14:50.225 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:50.225 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:14:50.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:50.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:51.236 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1966,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:14:51.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:51.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:14:51.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:51.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:52.237 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1966,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:14:52.259 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:52.259 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:14:52.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:52.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:53.267 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1966,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:14:53.297 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:14:53.297 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:53.298 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:14:53.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:53.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:54.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:14:54.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:54.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:55.426 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:14:55.427 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:55.427 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:14:55.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:55.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:56.439 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:56.461 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:14:56.461 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:14:56.462 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:14:56.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:56.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:14:56.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:56.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:57.458 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1966,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:14:57.490 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:57.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:14:57.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:57.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:58.466 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1966,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:14:58.490 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:58.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:14:58.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:58.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:14:59.485 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1966,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:14:59.507 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:14:59.507 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:14:59.508 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:14:59.508 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:14:59.508 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:14:59.508 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:14:59.525 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:14:59.525 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:14:59.526 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:14:59.526 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:14:59.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:14:59.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:00.598 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:00.620 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:00.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:00.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:01.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:01.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:02.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:02.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:03.621 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:15:03.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:15:03.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:03.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:04.640 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:04.670 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:04.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:04.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:05.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:05.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:06.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:06.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:07.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:15:07.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:15:07.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:07.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:08.686 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:08.707 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1966,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:09.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:09.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:10.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:10.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:11.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:11.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:11.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:15:11.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:15:12.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:12.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:13.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:13.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:14.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:14.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:14.909 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1970,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:14.931 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1970,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:15:14.932 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:15:14.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:15:14.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:15:15.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:15.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:15.941 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1970,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:15:15.962 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:15:15.963 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:15:16.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:16.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:16.964 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1970,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:15:16.994 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:15:17.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:17.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:18.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:18.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:19.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:19.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:20.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:20.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:21.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:21.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:22.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:22.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:23.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:23.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:24.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:24.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:25.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:25.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:26.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:26.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:27.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:27.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:28.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:28.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:29.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:29.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:30.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:30.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:31.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:31.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:32.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:32.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:33.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:33.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:34.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:34.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:35.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:35.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:36.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:36.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:37.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:37.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:38.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:38.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:39.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:39.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:40.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:40.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:41.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:41.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:42.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:42.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:43.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:43.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:43.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:43.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:45.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:45.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:46.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:46.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:46.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:46.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:48.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:48.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:49.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:49.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:50.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:50.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:51.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:51.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:52.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:52.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:52.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:52.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:53.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:53.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:55.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:55.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:55.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:55.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:56.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:56.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:57.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:57.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:58.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:58.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:15:59.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:15:59.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:01.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:01.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:02.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:02.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:03.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:03.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:04.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:04.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:05.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:05.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:06.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:06.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:07.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:07.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:08.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:08.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:09.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:09.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:10.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:10.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:11.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:11.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:12.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:12.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:13.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:13.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:14.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:14.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:15.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:15.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:16.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:16.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:17.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:17.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:18.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:18.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:19.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:19.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:20.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:20.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:21.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:21.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:22.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:22.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:23.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:23.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:24.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:24.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:25.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:25.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:26.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:26.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:27.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:27.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:28.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:28.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:29.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:29.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:30.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:30.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:31.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:31.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:32.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:32.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:33.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:33.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:34.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:34.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:35.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:35.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:36.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:36.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:37.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:37.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:38.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:38.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:39.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:39.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:40.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:40.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:41.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:41.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:42.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:42.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:43.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:43.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:44.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:44.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:45.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:45.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:46.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:46.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:46.995 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:16:46.995 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:16:46.996 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:16:46.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:46.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:16:47.003 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:16:47.003 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:16:47.003 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:47.004 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:16:47.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:47.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:47.968 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1970,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:47.990 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1970,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:48.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:48.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:49.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:49.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:50.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:50.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:50.991 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:50.992 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:16:51.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:51.160+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:51.985 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1972,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:52.007 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1972,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:52.007 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:16:52.007 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:52.007 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:16:52.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:52.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:52.986 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1972,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:16:53.008 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:53.008 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:16:53.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:53.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:53.988 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1972,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:16:54.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:54.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:16:54.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:54.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:55.025 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1972,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:16:55.048 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:16:55.048 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:55.049 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:16:55.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:55.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:56.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:16:56.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:56.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:57.154 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:16:57.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:57.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:16:57.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:57.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:58.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:58.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:58.167 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1972,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:58.199 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1972,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:16:58.199 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:16:58.199 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:16:58.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:58.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:16:59.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:16:59.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:16:59.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1972,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:16:59.233 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:16:59.233 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:17:00.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:00.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:00.258 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1972,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:17:00.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:17:00.290 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:17:01.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:01.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:01.273 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1972,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:17:01.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:17:01.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T10:17:01.295 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:17:01.295 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:17:01.295 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:17:01.295 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:17:01.312 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:17:01.312 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:17:01.313 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:17:01.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:17:02.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:02.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:03.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:03.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:04.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:04.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:05.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:05.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:06.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:06.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:07.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:07.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:08.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:08.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:09.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:09.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:10.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:10.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:11.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:11.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:12.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:12.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:13.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:13.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:14.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:14.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:15.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:14.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:16.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:16.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:16.314 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:17:16.315 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:17:16.315 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:17:16.315 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:17:16.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:17:17.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:17.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:17.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:17.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:18.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:18.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:19.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:19.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:20.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:20.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:21.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:21.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:21.966 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1974,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:17:21.984 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1974,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:17:21.984 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:17:21.985 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:17:21.985 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:17:22.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:22.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:22.953 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1974,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:17:22.976 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:17:22.976 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:17:23.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:23.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:23.967 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1974,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:17:23.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:17:24.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:24.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:25.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:25.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:26.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:26.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:27.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:27.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:28.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:28.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:29.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:29.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:30.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:30.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:31.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:31.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:32.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:32.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:33.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:33.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:34.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:34.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:35.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:35.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:36.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:36.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:37.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:37.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:38.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:38.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:39.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:39.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:40.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:40.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:41.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:41.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:42.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:42.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:43.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:43.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:44.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:44.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:45.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:45.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:46.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:46.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:47.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:47.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:48.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:48.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:49.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:49.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:50.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:50.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:51.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:51.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:52.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:52.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:53.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:53.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:54.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:54.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:55.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:55.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:56.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:56.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:57.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:57.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:58.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:58.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:17:59.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:17:59.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:00.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:00.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:01.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:01.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:02.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:02.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:03.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:03.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:04.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:04.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:05.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:05.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:06.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:06.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:07.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:07.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:08.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:08.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:09.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:09.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:10.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:10.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:11.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:11.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:12.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:12.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:13.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:13.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:14.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:14.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:15.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:15.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:16.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:16.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:17.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:17.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:18.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:18.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:19.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:19.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:20.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:20.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:21.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:21.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:22.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:22.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:23.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:23.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:24.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:24.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:25.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:25.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:26.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:26.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:27.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:27.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:28.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:28.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:29.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:29.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:30.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:30.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:31.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:31.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:32.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:32.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:33.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:33.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:34.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:34.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:35.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:35.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:36.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:36.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:37.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:37.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:38.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:38.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:39.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:39.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:40.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:40.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:41.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:41.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:42.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:42.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:43.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:43.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:44.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:44.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:45.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:45.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:46.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:46.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:47.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:47.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:48.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:48.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:49.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:49.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:50.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:50.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:51.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:51.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:52.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:52.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:53.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:53.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:53.999 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:18:53.999 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:18:53.999 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:18:54.000 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:18:54.000 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:18:54.005 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:18:54.005 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:18:54.006 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:18:54.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:18:54.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:54.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:55.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:55.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:56.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:56.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:57.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:57.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:58.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:58.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:18:59.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:18:59.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:00.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:00.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:01.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:01.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:02.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:02.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:03.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:03.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:04.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:04.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:05.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:05.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:06.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:06.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:07.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:07.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:08.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:08.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:09.007 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:19:09.007 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:19:09.007 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:19:09.008 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:09.008 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:09.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:09.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:10.007 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:10.028 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:10.028 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:19:10.029 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:10.029 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:19:10.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:10.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:10.963 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1978,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:10.995 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:10.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:19:11.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:11.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:11.982 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1978,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:19:11.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:11.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:19:12.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:12.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:12.961 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1978,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:12.983 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:19:12.983 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:12.983 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:19:13.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:13.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:14.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:19:14.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:14.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:15.077 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:19:15.077 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:15.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:15.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:15.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:16.071 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:16.093 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:16.093 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:19:16.093 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:19:16.094 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:16.094 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:19:16.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:16.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:17.101 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1978,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:17.134 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:17.134 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:19:17.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:17.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:18.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1978,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:19:18.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:18.164 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:19:18.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:18.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:19.110 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1978,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:19.132 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:19:19.132 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:19:19.133 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:19:19.133 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:19:19.133 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:19:19.133 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:19:19.163 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:19:19.163 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:19:19.163 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:19.163 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:19.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:19.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:20.171 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:20.193 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:20.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:20.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:21.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:21.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:22.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:22.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:23.195 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:23.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:23.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:23.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:24.179 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:24.201 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:24.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:24.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:25.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:25.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:26.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:26.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:27.202 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:27.202 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:27.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:27.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:28.164 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:28.185 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1978,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:28.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:28.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:29.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:29.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:30.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:30.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:31.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:31.187 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:19:31.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:31.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:32.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:32.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:33.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:33.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:33.865 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:33.901 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:19:33.902 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:19:33.902 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:33.902 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:19:34.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:34.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:34.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1982,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:34.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:19:34.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:19:35.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:35.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:35.892 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1982,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:19:35.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:19:36.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:36.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:37.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:37.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:38.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:38.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:39.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:39.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:40.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:40.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:41.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:41.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:42.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:42.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:43.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:43.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:44.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:44.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:45.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:45.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:46.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:46.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:47.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:47.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:48.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:48.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:49.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:49.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:50.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:50.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:51.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:51.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:52.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:52.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:53.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:53.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:54.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:54.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:55.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:55.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:56.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:56.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:57.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:57.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:58.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:58.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:19:59.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:19:59.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:00.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:00.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:01.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:01.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:02.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:02.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:03.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:03.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:04.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:04.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:05.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:05.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:06.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:06.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:07.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:07.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:08.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:08.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:09.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:09.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:10.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:10.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:11.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:11.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:13.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:13.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:14.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:14.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:15.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:15.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:16.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:16.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:17.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:16.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:17.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:17.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:18.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:18.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:19.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:19.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:20.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:20.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:21.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:21.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:22.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:22.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:23.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:23.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:24.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:24.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:25.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:25.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:26.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:26.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:27.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:27.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:28.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:28.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:29.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:29.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:30.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:30.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:31.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:31.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:32.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:32.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:33.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:33.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:34.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:34.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:35.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:35.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:36.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:36.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:37.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:37.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:38.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:38.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:39.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:39.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:40.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:40.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:41.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:41.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:42.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:42.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:43.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:43.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:44.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:44.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:45.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:45.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:46.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:46.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:47.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:47.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:48.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:48.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:49.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:49.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:50.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:50.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:51.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:51.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:52.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:52.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:53.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:53.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:54.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:54.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:55.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:55.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:56.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:56.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:57.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:57.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:58.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:58.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:20:59.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:20:59.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:00.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:00.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:01.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:01.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:02.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:02.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:03.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:03.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:04.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:04.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:05.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:05.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:05.924 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:21:05.924 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:21:05.924 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:21:05.924 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:05.925 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:21:05.932 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:21:05.932 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:21:05.932 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:05.932 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:21:06.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:06.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:06.933 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:06.965 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1982,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:07.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:07.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:08.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:08.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:09.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:09.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:09.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:09.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:21:10.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:10.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:11.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:11.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:12.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:12.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:13.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:13.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:14.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:14.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:14.817 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:14.839 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:14.839 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:21:14.840 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:14.840 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:21:15.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:15.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:15.859 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1988,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:15.881 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:15.881 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:21:16.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:16.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:17.110 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1988,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:17.140 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:17.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:21:17.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:17.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:18.140 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1988,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:18.164 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:21:18.164 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:18.165 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:21:18.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:18.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:19.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:21:19.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:19.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:20.243 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:21:20.243 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:20.244 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:21:20.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:20.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:21.124 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:21.154 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:21.154 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:21:21.154 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:21:21.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:21.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:21:21.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:21.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:22.166 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1988,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:22.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:22.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:21:22.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:22.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:23.183 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1988,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:23.215 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:23.215 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:21:23.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:23.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:24.190 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1988,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:24.212 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:21:24.212 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:21:24.212 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:21:24.212 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:21:24.212 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:21:24.212 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:21:24.233 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:21:24.233 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:21:24.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:24.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:21:24.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:24.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:25.304 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:25.325 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1988,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:25.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:25.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:26.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:26.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:27.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:27.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:28.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:28.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:21:28.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:28.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:29.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:29.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:30.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:30.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:31.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:31.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:32.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:32.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:33.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:33.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:34.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:34.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:35.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:35.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:36.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:36.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:37.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:37.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:38.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:38.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:39.052 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1992,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:39.081 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1992,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:21:39.082 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:21:39.082 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:39.082 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:21:39.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:39.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:40.094 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1992,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:40.126 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:21:40.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:21:40.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:40.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:41.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1992,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:21:41.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:21:41.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:41.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:42.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:42.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:43.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:43.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:44.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:44.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:45.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:45.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:46.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:46.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:47.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:47.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:48.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:48.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:49.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:49.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:50.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:50.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:51.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:51.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:52.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:52.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:53.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:53.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:54.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:54.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:55.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:55.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:56.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:56.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:57.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:57.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:58.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:58.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:21:59.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:21:59.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:00.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:00.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:01.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:01.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:02.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:02.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:03.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:03.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:04.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:04.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:05.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:05.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:06.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:06.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:07.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:07.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:08.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:08.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:09.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:09.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:10.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:10.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:11.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:11.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:12.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:12.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:13.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:13.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:14.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:14.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:15.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:15.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:16.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:16.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:17.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:17.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:18.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:18.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:19.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:19.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:20.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:20.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:21.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:21.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:22.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:22.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:23.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:23.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:24.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:24.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:25.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:25.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:26.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:26.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:27.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:27.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:28.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:28.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:29.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:29.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:30.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:30.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:31.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:31.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:32.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:32.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:33.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:33.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:34.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:34.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:35.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:35.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:36.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:36.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:37.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:37.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:38.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:38.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:39.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:39.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:40.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:40.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:41.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:41.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:42.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:42.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:43.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:43.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:44.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:44.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:45.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:45.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:46.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:46.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:47.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:47.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:48.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:48.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:49.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:49.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:50.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:50.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:51.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:51.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:52.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:52.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:53.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:53.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:54.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:54.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:55.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:55.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:56.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:56.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:57.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:57.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:58.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:58.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:22:59.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:22:59.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:00.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:00.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:01.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:01.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:02.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:02.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:03.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:03.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:04.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:04.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:05.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:05.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:06.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:06.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:07.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:07.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:08.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:08.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:09.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:09.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:10.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:10.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:11.154 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:23:11.155 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:23:11.155 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:23:11.155 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:11.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:23:11.161 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:23:11.161 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:23:11.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:11.161 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:11.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:11.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:12.176 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1992,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:12.195 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1992,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:12.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:12.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:13.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:13.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:14.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:14.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:15.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:15.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:15.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:15.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:16.204 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:16.235 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:16.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:23:16.236 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:16.236 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:23:16.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:16.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:17.249 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1994,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:17.271 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:17.272 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:23:17.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:17.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:18.269 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1994,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:18.301 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:18.301 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:23:18.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:18.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:19.295 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1994,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:19.317 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:23:19.318 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:19.318 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:23:19.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:19.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:20.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:20.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:20.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:23:21.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:21.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:21.562 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:23:21.562 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:21.563 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:22.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:22.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:22.530 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:22.552 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:22.553 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:23:22.553 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:23:22.553 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:22.553 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:23:23.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:23.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:23.553 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":1994,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:23.581 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:23.582 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:23:24.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:24.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:24.575 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1994,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:24.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:24.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:23:25.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:25.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:25.641 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":1994,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:25.666 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:23:25.666 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:23:25.666 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:23:25.666 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:23:25.666 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:23:25.667 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:23:25.700 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:23:25.701 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:23:25.701 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:25.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:26.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:26.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:26.662 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:26.700 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:27.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:27.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:28.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:28.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:29.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:29.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:29.701 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:29.701 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:30.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:30.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:30.719 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:30.742 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:31.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:31.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:32.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:32.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:33.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:33.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:33.743 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:33.743 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:34.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:34.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:34.716 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:34.747 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1994,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:35.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:35.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:36.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:36.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:37.426 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:37.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:37.747 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:37.748 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:23:38.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:38.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:39.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:39.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:40.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:40.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:41.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:41.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:42.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:42.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:42.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:42.557 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:23:42.558 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:23:42.558 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:42.558 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:23:43.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:43.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:43.584 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":1996,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:43.614 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:23:43.615 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:23:44.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:44.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:44.629 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":1996,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:23:44.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:23:45.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:45.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:46.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:46.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:47.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:47.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:48.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:48.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:49.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:49.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:50.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:50.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:51.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:51.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:52.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:52.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:53.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:53.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:54.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:54.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:55.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:55.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:56.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:56.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:57.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:57.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:58.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:58.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:23:59.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:23:59.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:00.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:00.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:01.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:01.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:02.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:02.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:03.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:03.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:04.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:04.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:05.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:05.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:06.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:06.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:07.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:07.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:08.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:08.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:09.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:09.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:10.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:10.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:11.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:11.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:12.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:12.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:13.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:13.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:14.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:14.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:15.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:15.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:16.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:16.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:17.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:17.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:18.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:18.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:19.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:19.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:20.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:20.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:21.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:21.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:22.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:22.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:23.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:23.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:24.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:24.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:25.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:25.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:26.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:26.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:27.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:27.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:28.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:28.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:29.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:29.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:30.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:30.402+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:31.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:31.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:32.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:32.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:33.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:33.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:34.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:34.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:35.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:35.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:36.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:36.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:37.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:37.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:38.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:38.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:39.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:39.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:40.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:40.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:41.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:41.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:42.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:42.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:43.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:43.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:44.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:44.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:45.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:45.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:46.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:46.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:47.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:47.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:48.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:48.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:49.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:49.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:50.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:50.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:51.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:51.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:52.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:52.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:53.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:53.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:54.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:54.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:55.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:55.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:56.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:56.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:57.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:57.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:58.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:58.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:24:59.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:24:59.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:00.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:00.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:01.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:01.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:02.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:02.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:03.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:03.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:04.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:04.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:05.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:05.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:06.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:06.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:07.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:07.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:08.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:08.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:09.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:09.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:10.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:10.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:11.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:11.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:12.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:12.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:13.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:13.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:14.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:14.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:14.653 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:25:14.653 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:25:14.653 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:25:14.654 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:25:14.654 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:25:14.661 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:25:14.661 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:25:14.662 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:14.662 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:15.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:15.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:15.682 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:15.715 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:16.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:16.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:17.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:17.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:18.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:18.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:18.725 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:18.726 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:19.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:19.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:19.718 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:19.741 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":1996,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:20.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:20.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:21.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:21.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:22.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:22.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:22.742 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:22.743 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:23.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:23.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:24.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:24.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:25.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:25.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:26.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:26.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:27.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:27.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:27.602 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2000,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:27.630 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2000,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:28.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:28.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:29.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:29.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:30.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:30.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:30.631 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:30.631 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:31.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:31.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:31.649 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2000,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:31.672 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2000,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":4,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:32.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:32.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:33.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:33.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:34.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:34.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:34.673 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:34.674 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:35.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:35.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:35.671 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:35.694 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:35.694 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:25:35.694 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:35.694 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:25:36.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:36.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:36.701 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2004,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:25:36.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:36.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:25:37.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:37.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:37.743 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2004,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:25:37.769 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:37.769 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:25:38.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:38.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:38.734 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2004,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:25:38.764 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:25:38.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:38.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:25:39.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:39.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:39.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:25:40.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:40.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:40.889 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:25:40.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:40.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:41.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:41.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:41.892 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:41.915 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:41.915 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:25:41.916 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:25:41.916 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:41.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:25:42.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:42.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:42.893 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2004,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:25:42.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:42.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:25:43.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:43.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:43.916 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2004,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:25:43.939 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:43.939 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:25:44.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:44.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:44.922 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2004,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:25:44.945 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:25:44.945 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:25:44.946 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:25:44.946 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:25:44.946 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:25:44.946 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:25:44.966 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:25:44.966 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:25:44.966 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:44.966 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:45.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:45.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:45.988 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:46.011 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:46.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:46.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:47.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:47.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:48.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:48.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:49.011 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:49.012 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:49.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:49.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:49.974 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:49.997 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:50.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:50.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:51.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:51.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:52.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:52.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:52.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:52.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:53.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:53.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:54.013 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:54.041 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2004,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:25:54.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:54.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:55.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:55.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:56.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:56.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:57.041 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:25:57.043 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:25:57.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:57.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:58.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:58.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:25:59.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:25:59.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:00.468 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2008,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:26:00.490 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2008,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:26:00.491 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:26:00.491 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:26:00.491 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:26:00.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:00.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:01.505 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2008,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:26:01.536 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:26:01.536 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:26:01.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:01.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:02.533 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2008,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:26:02.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:26:02.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:02.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:03.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:03.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:04.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:04.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:05.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:05.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:06.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:06.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:07.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:07.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:08.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:08.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:09.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:09.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:10.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:10.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:11.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:11.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:12.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:12.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:13.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:13.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:14.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:14.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:15.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:15.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:16.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:16.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:17.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:17.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:18.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:18.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:19.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:19.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:20.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:20.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:21.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:21.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:22.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:22.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:23.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:23.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:24.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:24.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:25.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:25.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:26.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:26.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:27.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:27.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:28.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:28.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:29.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:29.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:30.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:30.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:31.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:31.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:32.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:32.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:33.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:33.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:34.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:34.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:35.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:35.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:36.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:36.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:37.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:37.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:38.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:38.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:39.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:39.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:40.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:40.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:41.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:41.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:42.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:42.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:43.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:43.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:44.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:44.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:45.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:45.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:46.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:46.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:47.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:47.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:48.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:48.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:49.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:49.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:50.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:50.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:51.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:51.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:52.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:52.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:53.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:53.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:54.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:54.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:55.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:55.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:56.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:56.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:57.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:57.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:58.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:58.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:26:59.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:26:59.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:00.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:00.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:01.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:01.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:02.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:02.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:03.472 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:03.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:04.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:04.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:05.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:05.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:06.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:06.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:07.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:07.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:08.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:08.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:09.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:09.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:10.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:10.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:11.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:11.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:12.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:12.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:13.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:13.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:14.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:14.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:15.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:15.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:16.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:16.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:17.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:17.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:18.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:18.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:19.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:19.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:20.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:20.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:21.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:21.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:22.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:22.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:23.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:23.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:24.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:24.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:25.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:25.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:26.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:26.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:27.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:27.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:28.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:28.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:29.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:29.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:30.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:30.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:31.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:31.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:32.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:32.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:32.559 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:27:32.559 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:27:32.560 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:27:32.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:32.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:27:32.566 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:27:32.566 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:27:32.567 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:32.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:27:33.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:33.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:33.541 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2008,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:33.564 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2008,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:34.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:34.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:35.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:35.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:36.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:36.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:36.565 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:36.565 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:27:37.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:37.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:38.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:38.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:39.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:39.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:40.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:40.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:41.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:41.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:41.477 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:41.502 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2012,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:42.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:42.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:43.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:43.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:44.503 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:44.503 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:27:44.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:44.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:45.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2014,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:45.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:45.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:45.560 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2014,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:45.560 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:27:45.561 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:45.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:27:46.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:46.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:46.606 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2014,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:27:46.629 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:46.629 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:27:47.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:47.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:47.602 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2014,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:27:47.624 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:47.624 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:27:48.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:48.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:48.627 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2014,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:27:48.648 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:27:48.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:48.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:27:49.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:49.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:49.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:27:50.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:50.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:50.774 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:27:50.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:50.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:27:51.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:51.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:51.803 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2014,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:51.832 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2014,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:27:51.833 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:27:51.833 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:27:51.833 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:51.834 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:27:52.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:52.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:52.842 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2014,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:27:52.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:52.865 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:27:53.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:53.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:53.805 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2014,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:27:53.827 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:27:53.828 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:27:54.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:54.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:54.790 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2014,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:27:54.812 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:27:54.813 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T10:27:54.813 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:27:54.813 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:27:54.813 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:27:54.813 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:27:54.829 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:27:54.829 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T10:27:54.830 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T10:27:54.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:27:55.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:55.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:56.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:56.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:57.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:57.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:58.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:58.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:27:59.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:27:59.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:00.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:00.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:01.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:01.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:02.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:02.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:03.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:03.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:04.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:04.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:05.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:05.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:06.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:06.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:07.437 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:07.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:08.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:08.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:09.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:09.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:09.831 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T10:28:09.832 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T10:28:09.832 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:28:09.832 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:28:09.832 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:28:10.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:10.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:11.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:11.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:12.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:12.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:13.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:13.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:14.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:14.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:15.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:15.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:15.497 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2018,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:28:15.528 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2018,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:28:15.528 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:28:15.528 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:28:15.528 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:28:16.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:16.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:16.491 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2018,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:28:16.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:28:16.514 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:28:17.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:17.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:17.508 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2018,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:28:17.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:28:18.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:18.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:19.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:19.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:20.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:20.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:21.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:21.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:22.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:22.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:23.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:23.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:24.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:24.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:25.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:25.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:26.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:26.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:27.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:27.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:28.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:28.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:29.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:29.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:30.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:30.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:31.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:31.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:32.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:32.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:33.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:33.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:34.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:34.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:35.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:35.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:36.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:36.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:37.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:37.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:38.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:38.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:39.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:39.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:40.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:40.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:41.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:41.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:42.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:42.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:43.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:43.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:44.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:44.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:45.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:45.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:46.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:46.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 160 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:47.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:47.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:48.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:48.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:49.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:49.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:50.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:50.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:51.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:51.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:52.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:52.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:53.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:53.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:54.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:54.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:55.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:55.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:56.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:56.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:57.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:57.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:58.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:58.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:28:59.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:28:59.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:00.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:00.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:01.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:01.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:02.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:02.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:03.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:03.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:04.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:04.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:05.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:05.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:06.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:06.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:07.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:07.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:08.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:08.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:09.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:09.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:10.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:10.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:11.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:11.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:12.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:12.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:13.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:13.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:14.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:14.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:15.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:15.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:16.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:16.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:17.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:17.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:18.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:18.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:19.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:19.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:20.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:20.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:21.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:21.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:22.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:22.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:23.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:23.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:24.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:24.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:25.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:25.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:26.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:26.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:27.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:27.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:28.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:28.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:29.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:29.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:30.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:30.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:31.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:31.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:32.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:32.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:33.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:33.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:34.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:34.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:35.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:35.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:36.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:36.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:37.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:37.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:38.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:38.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:39.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:39.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:40.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:40.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:41.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:41.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:42.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:42.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:43.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:43.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:44.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:44.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:45.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:45.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:46.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:46.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:47.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:47.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:47.529 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:29:47.530 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:29:47.530 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:29:47.530 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:29:47.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:29:47.536 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:29:47.536 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T10:29:47.537 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T10:29:47.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:29:48.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:48.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:49.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:49.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:50.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:50.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:51.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:51.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:52.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:52.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:53.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:53.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:54.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:54.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:55.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:55.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:56.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:56.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:57.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:57.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:58.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:58.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:29:59.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:29:59.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:00.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:00.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:01.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:01.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:02.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:02.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:02.537 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T10:30:02.538 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T10:30:02.538 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:30:02.538 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:02.539 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:30:03.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:03.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:03.527 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:03.549 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:03.549 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:30:03.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:03.550 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:30:04.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:04.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:04.522 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2022,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:04.554 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:04.554 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:30:05.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:05.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:05.528 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2022,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:05.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:05.561 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:30:06.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:06.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:06.550 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2022,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:06.580 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:30:06.580 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:06.580 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:30:07.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:07.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:07.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:30:08.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:08.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:08.730 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:30:08.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:08.731 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:30:09.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:09.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:09.682 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:09.713 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:09.714 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:30:09.714 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:30:09.714 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:09.714 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:30:10.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:10.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:10.672 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2022,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:10.703 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:10.703 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:30:11.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:11.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:11.682 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2022,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:11.706 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:11.706 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:30:12.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:12.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:12.698 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2022,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:12.721 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:30:12.721 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:30:12.721 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:30:12.721 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:30:12.721 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:30:12.722 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:30:12.732 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:30:12.733 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:30:12.733 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:12.733 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:30:13.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:13.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:13.808 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:13.837 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:14.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:14.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:15.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:15.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:16.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:16.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:16.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:16.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:30:17.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:17.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:17.839 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:17.861 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2022,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:18.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:18.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:19.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:19.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:20.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:20.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:20.862 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:20.863 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:30:21.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:21.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:22.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:22.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:23.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:23.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:24.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:24.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:25.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:25.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:26.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:26.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:27.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:27.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:28.248 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:28.270 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:30:28.271 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:30:28.271 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:28.271 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:30:28.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:28.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:29.239 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2026,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:29.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:29.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:29.270 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:30:29.271 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:30:30.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:30.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:30.327 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2026,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:30:30.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:30:31.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:31.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:32.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:32.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:33.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:33.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:34.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:34.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:35.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:35.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:36.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:36.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:37.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:37.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:38.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:38.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:39.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:39.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:40.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:40.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:41.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:41.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:42.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:42.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:43.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:43.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:44.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:44.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:45.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:45.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:46.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:46.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:47.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:47.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:48.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:48.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:49.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:49.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:50.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:50.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:51.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:51.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:52.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:52.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:53.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:53.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:54.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:54.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:55.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:55.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:56.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:56.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:57.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:57.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:58.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:58.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:30:59.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:30:59.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:00.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:00.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:01.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:01.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:02.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:02.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:03.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:03.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:04.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:04.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:05.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:05.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:06.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:06.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:07.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:07.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:08.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:08.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:09.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:09.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:10.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:10.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:11.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:11.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:12.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:12.674+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:13.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:13.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:14.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:14.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:15.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:15.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:16.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:16.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:17.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:17.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:18.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:18.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:19.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:19.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:20.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:20.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:21.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:21.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:22.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:22.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:23.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:23.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:24.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:24.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:25.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:25.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:26.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:26.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:27.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:27.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:28.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:28.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:29.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:29.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:30.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:30.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:31.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:31.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:32.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:32.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:33.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:33.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:34.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:34.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:35.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:35.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:36.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:36.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:37.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:37.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:38.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:38.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:39.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:39.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:40.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:40.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:41.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:41.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:42.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:42.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:43.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:43.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:44.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:44.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:45.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:45.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:46.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:46.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:47.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:47.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:48.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:48.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:49.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:49.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:50.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:50.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:51.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:51.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:52.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:52.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:53.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:53.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:54.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:54.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:55.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:55.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:56.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:56.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:57.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:57.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:58.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:58.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:31:59.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:31:59.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:00.350 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:32:00.350 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:32:00.350 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:32:00.350 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:00.350 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:32:00.358 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:32:00.358 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:32:00.358 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:00.359 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:00.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:00.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:01.343 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:01.366 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2026,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:01.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:01.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:02.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:02.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:03.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:03.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:04.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:04.369 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:04.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:04.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:05.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:05.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:06.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:06.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:07.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:07.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:08.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:08.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:09.260 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:09.292 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:09.292 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:32:09.293 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:09.293 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:32:09.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:09.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:10.280 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2032,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:10.312 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:10.312 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:32:10.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:10.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:11.543 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2032,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:11.575 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:11.575 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:32:11.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:11.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:12.556 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2032,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:12.578 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:32:12.579 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:12.579 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:32:12.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:12.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:13.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:13.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:13.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:32:14.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:14.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:14.691 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:32:14.691 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:14.691 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:15.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:15.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:15.707 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:15.732 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:15.732 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:32:15.732 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:32:15.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:15.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:32:16.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:16.669+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:16.746 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2032,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:16.769 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:16.769 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:32:17.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:17.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:17.827 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2032,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:17.849 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:17.849 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:32:18.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:18.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:18.808 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2032,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:18.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:32:18.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:32:18.831 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:32:18.831 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:32:18.832 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:32:18.832 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:32:18.848 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:32:18.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:32:18.849 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:18.849 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:19.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:19.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:19.904 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:19.926 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:20.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:20.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:21.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:21.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:22.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:22.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:22.927 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:22.927 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:23.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:23.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:23.913 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:23.945 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:24.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:24.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:25.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:25.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:26.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:26.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:26.947 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:26.947 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:27.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:27.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:27.930 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:27.956 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2032,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:28.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:28.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:29.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:29.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:30.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:30.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:30.957 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:30.958 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:32:31.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:31.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:32.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:32.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:33.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:33.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:34.276 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2036,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:34.298 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2036,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:32:34.298 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:32:34.298 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:34.299 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:32:34.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:34.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:35.292 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2036,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:35.315 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:32:35.315 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:32:35.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:35.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:36.330 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2036,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:32:36.352 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:32:36.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:36.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:37.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:37.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:38.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:38.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:39.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:39.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:40.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:40.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:41.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:41.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:42.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:42.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:43.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:43.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:44.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:44.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:45.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:45.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:46.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:46.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:48.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:48.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:48.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:48.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:49.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:49.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:50.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:50.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:51.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:51.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:52.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:52.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:54.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:54.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:54.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:54.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:55.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:55.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:57.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:57.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:58.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:58.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:32:59.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:32:59.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:00.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:00.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:01.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:01.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:02.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:02.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:03.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:03.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:04.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:04.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:05.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:05.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:06.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:06.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:07.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:07.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:08.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:08.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:09.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:09.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:10.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:10.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:11.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:11.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:12.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:12.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:13.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:13.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:14.244 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:14.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:15.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:15.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:16.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:16.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:17.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:17.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:18.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:18.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:19.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:19.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:20.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:20.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:21.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:21.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:22.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:22.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:23.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:23.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:24.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:24.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:25.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:25.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:26.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:26.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:27.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:27.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:28.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:28.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:29.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:29.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:30.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:30.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:31.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:31.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:32.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:32.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:33.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:33.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:34.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:34.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:35.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:35.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:36.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:36.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:37.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:37.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:38.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:38.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:39.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:39.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:40.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:40.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:41.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:41.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:42.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:42.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:43.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:43.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:44.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:44.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:45.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:45.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:46.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:46.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:47.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:47.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:48.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:48.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:49.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:49.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:50.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:50.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:51.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:51.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:52.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:52.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:53.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:53.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:54.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:54.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:55.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:55.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:56.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:56.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:57.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:57.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:58.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:58.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:33:59.273 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:33:59.271+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:00.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:00.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:01.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:01.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:02.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:02.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:03.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:03.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:04.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:04.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:05.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:05.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:06.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:06.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:06.353 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:34:06.354 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:34:06.354 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:34:06.354 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:06.354 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:34:06.362 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:34:06.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:34:06.362 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:06.363 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:34:07.326 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2036,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:07.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:07.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:07.349 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2036,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:08.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:08.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:09.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:09.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:10.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:10.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:10.350 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:10.350 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:34:11.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:11.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:11.337 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:11.362 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:11.363 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:34:11.363 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:11.363 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:34:12.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:12.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:12.367 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2038,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:12.389 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:12.389 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:34:13.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:13.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:13.367 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2038,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:13.389 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:13.390 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:34:14.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:14.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:14.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2038,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:14.428 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:34:14.428 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:14.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:34:15.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:15.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:15.565 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:34:16.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:16.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:16.567 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:34:16.567 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:16.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:34:17.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:17.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:17.569 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:17.599 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:17.599 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:34:17.599 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:34:17.600 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:17.600 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:34:18.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:18.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:18.556 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2038,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:18.586 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:18.586 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:34:19.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:19.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:19.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2038,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:19.620 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:19.620 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:34:20.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:20.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:20.628 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2038,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:20.650 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:34:20.650 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:34:20.651 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:34:20.651 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:34:20.651 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:34:20.651 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:34:20.671 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:34:20.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:34:20.671 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:20.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:34:21.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:21.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:21.768 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:21.798 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2038,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:22.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:22.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:23.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:23.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:24.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:24.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:24.800 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:24.800 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:34:25.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:25.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:26.475 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:26.473+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:27.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:27.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:28.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:28.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:29.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:29.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:30.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:30.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:31.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:31.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:32.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:32.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:33.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:33.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:34.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:34.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:35.484 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:35.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:35.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:35.514 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:34:35.514 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:34:35.514 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:35.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:34:36.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:36.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:36.512 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2042,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:36.535 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:34:36.535 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:34:37.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:37.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:37.536 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2042,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:34:37.558 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:34:38.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:38.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:39.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:39.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:40.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:40.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:41.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:41.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:42.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:42.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:43.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:43.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:44.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:44.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:45.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:45.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:46.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:46.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:47.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:47.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:48.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:48.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:49.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:49.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:50.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:50.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:51.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:51.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:52.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:52.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:53.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:53.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:54.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:54.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:55.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:55.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:56.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:56.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:57.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:57.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:58.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:58.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:34:59.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:34:59.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:00.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:00.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:01.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:01.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:02.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:02.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:03.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:03.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:04.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:04.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:05.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:05.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:06.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:06.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:07.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:07.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:08.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:08.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:09.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:09.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:10.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:10.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:11.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:11.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:12.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:12.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:13.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:13.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:14.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:14.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:15.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:15.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:16.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:16.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:17.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:17.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:18.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:18.668+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:19.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:19.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:20.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:20.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:21.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:21.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:22.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:22.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:23.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:23.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:24.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:24.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:25.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:25.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:26.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:26.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:27.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:27.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:28.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:28.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:29.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:29.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:30.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:30.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:31.592 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:31.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:32.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:32.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:33.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:33.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:34.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:34.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:35.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:35.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:36.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:36.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:37.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:37.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:38.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:38.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:39.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:39.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:40.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:40.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:41.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:41.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:42.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:42.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:43.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:43.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:44.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:44.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:45.658 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:45.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:46.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:46.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:47.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:47.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:48.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:48.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:49.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:49.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:50.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:50.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:51.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:51.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:52.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:52.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:53.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:53.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:54.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:54.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:55.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:55.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:56.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:56.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:57.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:57.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:58.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:58.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:35:59.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:35:59.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:00.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:00.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:01.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:01.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:02.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:02.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:03.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:03.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:04.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:04.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:05.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:05.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:06.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:06.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:07.559 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:36:07.559 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:36:07.559 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:36:07.560 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:07.560 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:36:07.566 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:36:07.566 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:36:07.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:07.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:07.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:07.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:08.549 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:08.566 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:08.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:08.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:09.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:09.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:10.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:10.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:11.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:11.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:11.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:11.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:12.560 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:12.592 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:12.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:12.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:13.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:13.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:14.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:14.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:15.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:15.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:15.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:15.594 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:16.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:16.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:16.594 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:16.612 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2042,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:17.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:17.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:18.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:18.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:19.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:19.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:19.614 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:19.614 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:20.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:20.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:20.621 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:20.641 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:20.641 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:36:20.641 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:20.641 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:36:21.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:21.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:21.641 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2044,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:21.671 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:21.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:36:22.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:22.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:22.693 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2044,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:22.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:22.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:36:23.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:23.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:23.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2044,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:23.724 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:36:23.725 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:23.725 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:36:24.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:24.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:24.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:36:25.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:25.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:25.866 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:36:25.866 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:25.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:26.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:26.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:26.850 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:26.871 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:26.871 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:36:26.872 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:36:26.872 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:26.872 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:36:27.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:27.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:27.923 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2044,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:27.952 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:27.952 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:36:28.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:28.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:28.973 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2044,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:29.003 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:29.003 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:36:29.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:29.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:30.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2044,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:30.031 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:36:30.031 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:36:30.031 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:36:30.031 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:36:30.032 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:36:30.032 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:36:30.069 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:36:30.069 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:36:30.070 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:30.070 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:30.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:30.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:31.001 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:31.019 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:31.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:31.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:32.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:32.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:33.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:33.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:34.020 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:34.021 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:34.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:34.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:35.020 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:35.043 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:35.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:35.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:36.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:36.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:37.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:37.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:38.044 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:38.044 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:38.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:38.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:39.029 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:39.047 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2044,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:39.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:39.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:40.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:40.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:41.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:41.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:42.048 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:42.048 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:36:42.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:42.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:43.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:43.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:44.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:44.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:45.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:45.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:46.095 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:46.127 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:36:46.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:36:46.127 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:46.128 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:36:46.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:46.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:47.136 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2046,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:36:47.159 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:36:47.159 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:36:47.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:47.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:48.159 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2046,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:36:48.182 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:36:48.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:48.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:49.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:49.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:50.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:50.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:51.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:51.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:52.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:52.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:53.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:53.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:54.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:54.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:55.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:55.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:56.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:56.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:57.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:57.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:58.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:58.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:36:59.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:36:59.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:00.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:00.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:01.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:01.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:02.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:02.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:03.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:03.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:04.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:04.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:05.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:05.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:06.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:06.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:07.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:07.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:08.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:08.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:09.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:09.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:10.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:10.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:11.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:11.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:12.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:12.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:13.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:13.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:14.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:14.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:15.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:15.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:16.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:16.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:17.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:17.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:18.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:18.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:19.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:19.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:20.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:20.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:21.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:21.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:22.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:22.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:23.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:23.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:24.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:24.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:25.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:25.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:26.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:26.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:27.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:27.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:28.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:28.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:29.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:29.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:30.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:30.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:31.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:31.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:32.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:32.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:33.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:33.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:34.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:34.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:35.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:35.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:36.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:36.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:37.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:37.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:38.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:38.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:39.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:39.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:40.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:40.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:41.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:41.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:42.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:42.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:43.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:43.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:44.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:44.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:45.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:45.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:46.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:46.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:47.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:47.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:48.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:48.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:49.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:49.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:50.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:50.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:51.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:51.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:52.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:52.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:53.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:53.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:54.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:54.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:55.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:55.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:56.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:56.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:57.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:57.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:58.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:58.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:37:59.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:37:59.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:00.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:00.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:01.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:01.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:02.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:02.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:03.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:03.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:04.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:04.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:05.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:05.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:06.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:06.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:07.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:07.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:08.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:08.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:09.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:09.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:10.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:10.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:11.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:11.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:12.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:12.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:13.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:13.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:14.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:14.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:15.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:15.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:16.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:16.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:17.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:17.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:18.183 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:38:18.183 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:38:18.183 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:38:18.183 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:38:18.183 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:38:18.193 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:38:18.193 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:38:18.193 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:18.193 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:38:18.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:18.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:19.201 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:19.224 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:19.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:19.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:20.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:20.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:21.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:21.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:22.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:22.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:38:22.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:22.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:23.286 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:23.318 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2046,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:23.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:23.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:24.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:24.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:25.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:25.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:26.319 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:26.319 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:38:26.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:26.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:27.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:27.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:28.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:28.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:29.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:29.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:30.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:30.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:31.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:31.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:32.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:32.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:33.512 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2054,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:33.530 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2054,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:33.530 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:38:33.530 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:33.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:38:33.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:33.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:34.496 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2054,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:38:34.518 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:34.519 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:38:35.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:35.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:35.497 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2054,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:38:35.527 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:35.527 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:38:35.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:35.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:36.485 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2054,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:38:36.503 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:38:36.503 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:36.504 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:38:37.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:37.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:37.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:38:38.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:38.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:38.629 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:38:38.630 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:38.630 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:38:39.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:39.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:39.619 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2054,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:39.642 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2054,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:38:39.642 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:38:39.642 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:38:39.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:39.643 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:38:40.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:40.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:40.637 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2054,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:38:40.665 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:40.666 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:38:41.212 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:41.210+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:41.659 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2054,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:38:41.690 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:41.691 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:38:42.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:42.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:42.688 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2054,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:38:42.711 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:38:42.711 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'c'] 2019-10-01T10:38:42.711 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:38:42.711 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:38:42.711 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:38:42.711 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:38:42.731 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:38:42.731 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T10:38:42.732 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T10:38:42.732 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:38:42.732 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:38:42.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:38:43.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:43.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:44.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:44.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:45.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:45.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:46.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:46.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:47.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:47.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:48.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:48.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:49.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:49.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:50.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:50.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:51.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:51.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:52.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:52.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:53.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:53.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:54.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:54.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:55.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:55.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:56.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:56.071+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:57.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:57.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:57.733 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T10:38:57.734 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T10:38:57.734 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:38:57.734 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:38:57.734 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:38:57.734 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:38:57.735 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:38:58.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:58.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:38:59.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:38:59.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:00.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:00.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:01.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:01.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:02.087 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:02.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:03.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:03.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:03.393 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2056,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:39:03.411 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2056,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:39:03.411 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:39:03.411 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:39:03.411 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:39:04.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:04.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:04.425 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2056,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:39:04.454 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:39:04.454 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:39:05.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:05.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:05.435 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2056,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:39:05.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:39:06.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:06.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:07.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:07.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:07.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:07.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:08.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:08.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:09.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:09.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:10.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:10.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:11.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:11.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:12.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:12.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:13.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:13.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:14.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:14.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:15.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:15.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:16.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:16.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:17.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:17.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:18.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:18.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:19.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:19.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:20.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:20.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:21.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:21.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:22.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:22.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:23.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:23.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:24.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:24.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:25.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:25.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:26.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:26.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:27.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:27.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:28.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:28.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:29.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:29.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:30.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:30.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:31.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:31.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:32.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:32.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:33.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:33.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:34.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:34.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:35.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:35.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:36.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:36.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:37.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:37.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:38.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:38.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:39.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:39.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:40.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:40.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:41.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:41.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:42.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:42.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:43.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:43.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:44.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:44.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:45.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:45.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:46.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:46.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:47.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:47.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:48.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:48.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:49.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:49.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:50.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:50.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:51.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:51.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:52.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:52.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:53.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:53.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:54.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:54.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:55.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:55.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:56.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:56.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:57.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:57.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:58.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:58.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:39:59.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:39:59.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:00.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:00.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:01.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:01.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:02.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:02.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:03.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:03.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:04.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:04.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:05.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:05.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:06.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:06.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:07.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:07.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:08.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:08.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:09.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:09.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:10.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:10.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:11.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:11.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:12.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:12.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:13.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:13.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:14.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:14.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:15.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:15.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:16.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:16.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:17.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:17.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:18.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:18.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:19.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:19.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:20.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:20.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:21.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:21.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:22.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:22.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:23.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:23.409+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:24.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:24.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:25.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:25.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:26.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:26.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:27.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:27.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:28.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:28.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:29.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:29.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:30.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:30.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:31.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:31.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:32.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:32.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:33.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:33.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:34.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:34.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:35.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:35.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:35.466 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:40:35.466 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:40:35.466 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:40:35.466 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:40:35.466 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:40:35.475 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:40:35.476 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T10:40:35.476 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T10:40:35.476 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:40:35.476 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:40:35.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:40:36.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:36.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:37.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:37.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:38.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:38.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:39.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:39.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:40.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:40.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:41.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:41.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:42.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:42.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:43.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:43.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:44.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:44.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:45.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:45.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:46.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:46.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:47.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:47.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:48.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:48.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:49.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:49.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:50.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:50.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:50.477 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T10:40:50.479 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T10:40:50.479 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:40:50.480 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:40:50.480 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:40:50.480 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:40:50.481 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:40:51.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:51.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:52.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:52.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:53.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:53.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:54.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:54.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:55.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:55.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:56.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:56.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:57.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:57.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:57.704 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:40:57.736 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:40:57.736 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:40:57.736 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:40:57.737 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:40:58.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:58.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:58.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2064,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:40:58.753 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:40:58.753 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:40:59.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:40:59.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:40:59.767 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2064,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:40:59.796 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:40:59.797 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:41:00.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:00.245+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:00.810 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2064,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:41:00.832 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:41:00.833 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:00.833 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:41:01.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:01.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:01.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:41:02.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:02.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:02.940 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:41:02.940 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:02.940 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:41:03.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:03.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:03.924 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:03.951 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:03.951 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:41:03.951 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:41:03.952 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:03.952 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:41:04.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:04.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:04.967 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2064,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:41:04.992 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:04.992 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:41:05.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:05.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:06.003 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2064,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:41:06.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:06.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:41:06.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:06.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:07.017 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2064,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:41:07.039 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:41:07.039 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:41:07.040 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:41:07.040 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:41:07.040 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:41:07.040 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:41:07.060 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:41:07.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:41:07.061 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:07.061 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:41:07.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:07.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:08.110 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:08.141 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:08.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:08.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:09.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:09.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:10.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:10.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:11.143 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:11.143 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:41:11.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:11.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:12.144 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:12.176 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:12.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:12.235+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:13.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:13.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:14.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:14.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:15.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:15.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:41:15.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:15.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:16.172 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:16.195 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2064,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:16.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:16.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:17.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:17.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:18.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:18.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:19.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:19.196 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:41:19.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:19.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:20.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:20.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:21.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:21.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:22.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:22.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:22.670 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:22.700 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:41:22.700 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:41:22.700 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:22.700 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:41:23.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:23.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:23.700 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2068,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:41:23.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:41:23.724 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:41:24.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:24.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:24.733 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2068,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:41:24.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:41:25.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:25.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:26.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:26.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:27.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:27.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:28.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:28.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:29.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:29.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:30.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:30.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:31.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:31.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:32.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:32.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:33.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:33.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:34.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:34.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:35.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:35.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:36.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:36.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:37.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:37.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:38.331 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:38.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:39.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:39.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:40.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:40.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:41.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:41.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:42.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:42.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:43.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:43.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:44.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:44.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:45.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:45.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:46.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:46.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:47.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:47.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:48.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:48.529+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:49.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:49.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:50.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:50.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:51.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:51.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:52.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:52.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:53.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:53.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:54.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:54.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:55.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:55.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:56.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:56.411+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:57.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:57.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:58.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:58.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:41:59.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:41:59.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:00.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:00.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:01.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:01.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:02.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:02.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:03.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:03.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:04.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:04.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:05.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:05.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:06.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:06.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:07.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:07.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:08.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:08.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:09.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:09.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:10.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:10.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:11.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:11.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:12.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:12.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:13.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:13.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:14.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:14.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:15.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:15.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:16.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:16.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:17.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:17.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:18.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:18.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:19.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:19.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:20.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:20.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:21.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:21.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:22.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:22.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:23.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:23.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:24.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:24.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:25.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:25.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:26.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:26.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:27.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:27.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:28.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:28.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:29.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:29.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:30.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:30.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:31.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:31.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:32.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:32.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:33.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:33.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:34.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:34.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:35.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:35.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:36.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:36.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:37.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:37.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:38.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:38.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:39.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:39.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:40.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:40.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:41.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:41.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:42.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:42.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:43.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:43.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:44.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:44.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:45.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:45.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:46.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:46.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:47.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:47.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:48.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:48.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:49.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:49.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:50.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:50.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:51.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:51.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:52.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:52.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:53.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:53.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:54.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:54.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:54.763 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:42:54.764 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:42:54.764 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:42:54.764 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:42:54.764 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:42:54.770 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:42:54.771 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:42:54.771 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:42:54.771 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:42:55.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:55.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:55.721 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:42:55.739 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:42:56.689 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:56.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:57.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:57.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:58.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:42:58.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:42:58.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:58.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:42:59.734 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:42:59.765 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:42:59.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:42:59.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:00.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:00.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:01.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:01.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:02.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:02.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:02.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:02.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:03.758 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:03.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:03.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:03.783 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2068,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:04.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:04.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:05.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:05.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:06.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:06.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:06.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:06.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:07.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:07.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:07.798 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:07.828 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:07.829 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:43:07.829 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:07.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:43:08.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:08.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:08.824 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2070,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:08.853 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:08.854 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:43:09.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:09.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:09.848 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2070,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:43:09.872 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:09.872 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:43:10.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:10.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:10.887 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2070,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:10.915 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:43:10.916 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:10.916 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:43:11.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:11.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:12.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:43:12.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:12.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:13.021 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:43:13.022 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:13.022 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:13.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:13.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:14.033 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:14.054 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:14.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:43:14.055 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:43:14.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:14.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:43:14.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:14.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:15.049 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2070,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:15.076 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:15.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:43:15.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:15.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:16.077 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2070,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T10:43:16.100 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:16.100 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:43:16.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:16.658+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:17.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2070,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:17.141 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:43:17.142 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:43:17.142 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:43:17.142 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:43:17.142 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:43:17.143 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:43:17.175 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:43:17.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:43:17.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:17.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:17.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:17.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:18.183 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:18.214 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:18.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:18.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:19.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:19.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:20.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:20.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:21.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:21.218 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:21.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:21.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:22.222 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:22.252 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2070,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:22.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:22.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:23.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:23.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:24.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:24.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:25.254 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:25.254 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:43:25.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:25.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:26.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:26.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:27.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:27.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:28.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:28.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:29.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:29.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:30.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:30.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:31.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:31.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:32.522 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:32.543 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:43:32.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:43:32.544 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:32.544 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:43:32.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:32.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:33.518 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2072,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:33.540 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:43:33.540 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:43:33.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:33.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:34.560 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2072,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:43:34.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:43:34.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:34.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:35.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:35.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:36.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:36.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:37.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:37.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:38.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:38.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:39.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:39.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:40.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:40.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:41.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:41.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:42.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:42.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:43.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:43.756+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:44.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:44.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:45.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:45.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:46.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:46.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:47.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:47.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:48.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:48.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:49.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:49.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:50.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:50.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:51.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:51.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:52.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:52.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:53.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:53.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:54.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:54.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:55.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:55.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:56.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:56.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:57.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:57.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:58.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:58.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:43:59.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:43:59.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:00.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:00.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:01.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:01.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:02.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:02.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:03.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:03.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:04.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:04.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:05.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:05.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:06.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:06.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:07.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:07.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:08.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:08.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:09.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:09.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:10.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:10.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:11.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:11.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:12.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:12.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:13.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:13.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:14.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:14.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:15.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:15.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:16.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:16.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:17.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:17.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:18.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:18.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:19.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:19.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:20.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:20.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:21.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:21.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:22.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:22.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:23.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:23.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:24.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:24.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:25.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:25.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:26.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:26.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:27.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:27.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:29.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:28.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:29.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:29.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:30.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:30.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:31.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:31.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:32.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:32.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:33.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:33.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:34.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:34.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:35.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:35.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:37.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:37.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:38.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:38.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:39.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:39.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:40.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:40.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:41.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:41.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:42.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:42.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:43.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:43.025+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:44.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:44.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:45.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:45.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:46.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:46.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:47.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:47.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:48.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:48.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:49.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:49.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:50.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:50.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:51.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:51.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:52.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:52.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:53.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:53.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:54.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:54.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:55.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:55.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:56.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:56.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:57.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:57.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:58.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:58.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:44:59.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:44:59.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:00.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:00.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:01.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:01.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:02.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:02.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:03.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:03.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:04.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:04.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:04.591 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:45:04.592 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:45:04.592 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:45:04.592 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:45:04.592 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:45:04.598 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:45:04.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:45:04.598 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:04.598 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:45:05.047 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:05.045+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:05.625 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:05.654 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2072,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:06.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:06.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:07.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:07.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:08.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:08.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:08.655 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:08.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:45:09.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:09.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:10.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:10.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:11.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:11.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:12.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:12.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:13.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:13.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:14.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:14.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:15.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:15.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:16.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:16.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:17.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:17.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:18.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:18.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:18.186 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:18.208 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:18.209 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:45:18.209 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:18.209 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:45:19.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:19.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:19.150 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2086,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:19.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:19.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:45:20.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:20.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:20.148 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2086,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:20.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:20.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:45:21.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:21.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:21.155 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2086,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:21.178 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:45:21.178 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:21.179 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:45:22.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:22.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:22.285 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:45:23.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:23.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:23.286 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:45:23.286 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:23.287 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:45:24.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:24.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:24.274 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:24.295 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:24.295 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:45:24.296 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:45:24.296 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:24.296 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:45:25.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:25.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:25.365 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2086,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:25.386 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:25.387 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:45:26.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:26.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:26.376 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2086,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:26.408 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:26.408 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:45:26.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:26.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:27.409 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2086,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:27.430 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:45:27.430 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:45:27.430 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:45:27.430 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:45:27.431 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:45:27.431 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:45:27.451 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:45:27.452 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:45:27.452 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:27.452 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:45:28.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:28.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:28.532 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:28.563 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2086,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:28.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:28.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:30.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:30.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:31.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:31.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:31.564 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:31.564 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:45:31.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:31.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:32.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:32.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:33.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:33.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:34.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:34.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:35.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:35.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:36.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:36.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:37.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:37.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:38.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:38.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:40.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:40.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:41.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:41.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:42.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:42.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:42.594 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2090,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:42.615 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2090,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:45:42.616 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:45:42.616 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:42.616 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:45:42.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:42.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:43.624 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2090,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:43.647 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:45:43.647 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:45:43.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:43.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:44.639 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2090,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:45:44.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:45:44.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:44.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:45.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:45.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:47.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:47.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:48.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:48.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:49.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:49.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:50.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:50.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:51.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:51.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:52.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:52.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:53.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:53.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:53.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:53.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:54.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:54.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:55.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:55.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:56.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:56.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:57.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:57.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:58.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:58.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:45:59.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:45:59.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:00.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:00.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:01.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:01.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:02.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:02.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:03.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:03.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:04.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:04.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:05.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:05.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:06.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:06.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:07.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:07.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:08.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:08.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:09.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:09.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:10.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:10.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:12.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:12.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:13.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:13.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:14.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:14.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:15.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:15.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:16.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:16.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:17.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:17.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:18.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:18.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:19.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:19.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:19.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:19.994+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:20.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:20.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:22.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:22.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:22.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:22.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:23.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:23.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:24.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:24.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:25.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:25.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:26.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:26.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:27.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:27.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:28.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:28.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:29.868 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:29.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:30.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:30.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:31.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:31.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:32.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:32.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:33.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:33.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:34.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:34.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:35.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:35.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:36.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:36.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:37.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:37.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:38.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:38.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:39.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:39.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:40.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:40.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:41.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:41.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:42.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:42.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:43.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:43.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:44.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:44.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:45.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:45.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:46.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:46.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:47.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:47.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:48.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:48.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:49.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:49.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:50.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:50.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:51.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:51.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:52.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:52.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:53.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:53.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:54.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:54.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:55.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:55.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:56.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:56.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:57.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:57.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:58.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:58.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:46:59.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:46:59.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:00.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:00.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:01.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:01.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:02.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:02.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:03.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:03.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:04.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:04.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:05.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:05.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:06.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:06.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:07.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:07.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:08.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:08.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:09.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:09.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:10.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:10.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:11.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:11.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:12.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:12.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:13.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:13.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:14.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:14.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:14.662 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:47:14.662 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:47:14.662 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:47:14.663 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:14.663 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:47:14.668 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:47:14.668 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:47:14.669 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:14.669 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:15.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:15.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:15.651 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2090,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:15.665 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2090,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:16.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:16.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:17.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:17.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:18.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:18.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:18.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:18.671 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:19.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:19.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:20.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:20.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:21.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:21.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:22.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:22.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:23.570 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:23.593 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:23.593 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:47:23.593 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:23.593 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:47:23.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:23.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:24.587 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2096,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:24.611 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:24.611 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:47:24.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:24.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:25.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:25.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:25.828 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2096,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:25.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:25.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:47:26.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:26.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:26.872 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2096,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:26.894 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:47:26.895 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:26.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:47:27.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:27.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:28.028 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:47:28.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:28.739+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:29.029 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:47:29.029 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:29.029 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:29.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:29.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:30.033 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:30.055 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:30.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:47:30.056 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:47:30.056 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:30.056 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:47:30.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:30.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:31.075 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2096,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:31.105 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:31.106 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:47:31.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:31.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:32.092 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2096,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:32.123 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:32.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:47:32.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:32.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:33.097 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2096,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:33.128 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:47:33.128 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:47:33.128 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:47:33.128 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:47:33.128 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:47:33.129 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:47:33.148 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:47:33.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:47:33.149 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:33.149 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:33.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:33.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:34.210 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:34.240 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:34.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:34.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:35.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:35.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:36.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:36.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:37.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:37.242 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:37.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:37.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:38.214 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:38.241 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:38.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:38.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:39.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:39.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:40.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:40.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:41.242 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:41.243 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:41.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:41.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:42.230 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:42.260 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2096,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:42.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:42.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:43.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:43.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:44.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:44.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:45.261 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:45.261 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:47:45.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:45.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:46.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:46.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:47.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:47.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:48.672 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2100,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:48.704 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2100,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:47:48.705 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:47:48.705 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:48.705 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:47:48.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:48.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:49.687 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2100,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:49.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:47:49.718 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:47:49.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:49.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:50.702 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2100,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:47:50.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:47:50.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:50.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:51.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:51.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:52.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:52.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:53.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:53.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:54.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:54.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:55.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:55.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:56.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:56.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:57.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:57.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:58.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:58.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:47:59.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:47:59.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:00.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:00.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:01.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:01.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:02.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:02.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:03.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:03.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:04.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:04.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:05.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:05.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:06.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:06.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:07.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:07.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:08.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:08.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:09.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:09.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:10.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:10.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:11.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:11.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:12.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:12.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:13.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:13.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:14.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:14.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:15.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:15.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:16.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:16.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:17.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:17.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:18.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:18.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:19.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:19.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:20.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:20.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:21.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:21.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:22.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:22.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:23.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:23.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:24.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:24.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:25.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:25.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:26.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:26.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:27.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:27.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:28.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:28.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:29.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:29.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:30.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:30.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:31.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:31.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:32.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:32.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:33.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:33.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:34.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:34.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:35.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:35.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:36.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:36.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:37.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:37.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:38.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:38.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:39.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:39.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:40.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:40.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:41.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:41.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:42.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:42.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:43.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:43.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:44.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:44.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:45.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:45.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:46.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:46.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:47.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:47.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:49.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:49.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:50.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:50.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:50.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:50.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:52.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:51.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:53.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:53.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:54.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:54.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:55.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:55.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:56.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:56.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:57.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:57.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:58.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:58.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:48:59.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:48:59.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:00.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:00.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:01.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:01.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:02.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:02.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:03.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:03.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:04.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:04.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:05.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:05.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:06.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:06.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:07.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:07.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:08.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:08.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:09.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:09.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:10.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:10.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:11.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:11.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:12.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:12.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:13.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:13.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:14.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:14.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:15.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:15.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:16.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:16.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:17.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:17.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:18.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:18.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:19.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:19.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:20.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:20.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:20.725 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:49:20.725 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:49:20.725 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:49:20.725 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:20.726 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:49:20.731 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:49:20.732 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:49:20.732 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:20.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:21.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:21.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:21.726 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2100,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:21.745 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2100,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:22.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:22.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:23.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:23.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:24.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:24.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:24.745 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:24.746 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:25.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:25.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:26.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:26.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:27.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:27.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:28.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:28.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:29.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:29.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:29.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:29.678 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:29.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:49:29.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:29.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:49:30.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:30.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:30.675 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2106,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:30.707 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:30.707 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:49:31.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:31.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:31.679 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2106,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:31.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:31.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:49:32.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:32.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:32.719 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2106,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:32.740 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:49:32.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:32.741 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:49:33.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:33.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:33.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:49:34.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:34.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:34.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:49:34.868 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:34.868 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:35.158 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:35.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:35.819 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:35.842 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:35.842 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:49:35.842 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:49:35.843 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:35.843 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:49:36.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:36.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:36.801 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2106,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:36.823 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:36.823 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:49:37.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:37.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:37.828 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2106,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:37.860 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:37.860 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:49:38.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:38.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:38.875 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2106,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:38.894 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:49:38.894 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:49:38.894 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:49:38.894 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:49:38.895 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:49:38.895 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:49:38.913 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:49:38.914 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:49:38.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:38.914 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:39.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:39.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:39.989 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:40.019 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:40.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:40.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:41.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:41.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:42.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:42.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:43.020 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:43.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:43.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:43.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:44.015 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:44.037 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2106,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:44.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:44.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:45.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:45.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:46.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:46.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:47.038 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:47.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:49:47.252 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:47.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:48.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:48.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:49.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:49.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:50.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:50.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:51.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:51.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:52.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:52.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:53.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:53.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:53.753 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2110,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:53.774 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2110,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:49:53.774 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:49:53.775 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:53.775 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:49:54.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:54.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:54.732 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2110,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:54.755 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:49:54.755 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:49:55.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:55.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:55.718 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2110,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:49:55.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:49:56.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:56.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:57.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:57.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:58.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:58.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:49:59.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:49:59.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:00.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:00.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:01.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:01.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:02.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:02.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:03.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:03.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:04.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:04.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:05.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:05.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:06.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:06.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:07.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:07.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:08.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:08.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:09.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:09.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:10.232 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:10.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:11.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:11.230+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:12.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:12.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:13.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:13.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:14.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:14.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:15.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:15.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:16.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:16.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:17.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:17.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:18.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:18.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:19.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:19.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:20.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:20.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:21.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:21.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:22.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:22.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:23.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:23.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:24.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:24.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:25.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:25.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:26.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:26.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:27.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:27.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:28.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:28.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:29.162 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:29.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:30.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:30.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:31.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:31.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:32.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:32.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:33.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:33.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:34.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:34.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:35.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:35.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:36.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:36.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:37.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:37.242+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:38.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:38.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:39.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:39.250+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:40.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:40.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:41.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:41.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:42.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:42.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:43.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:43.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:44.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:44.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:45.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:45.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:46.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:46.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:47.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:47.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:48.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:48.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:49.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:49.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:50.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:50.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:51.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:51.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:52.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:52.157+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:53.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:53.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:54.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:54.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:55.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:55.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:56.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:56.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:57.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:57.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:58.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:58.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:50:59.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:50:59.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:00.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:00.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:01.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:01.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:02.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:02.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:03.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:03.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:04.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:04.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:05.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:05.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:06.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:06.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:07.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:07.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:08.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:08.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:09.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:09.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:10.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:10.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:11.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:11.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:12.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:12.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:13.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:13.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:14.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:14.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:15.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:15.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:16.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:16.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:17.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:17.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:18.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:18.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:19.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:19.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:20.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:20.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:21.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:21.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:22.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:22.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:23.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:23.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:24.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:24.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:25.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:25.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:25.741 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:51:25.741 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T10:51:25.741 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T10:51:25.741 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:25.742 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T10:51:25.751 INFO:tasks.ceph.mon.c:Started 2019-10-01T10:51:25.751 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:51:25.751 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:25.752 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:51:26.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:26.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:26.739 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2110,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:26.760 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2110,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:27.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:27.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:28.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:28.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:29.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:29.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:29.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:29.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:51:30.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:30.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:31.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:31.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:32.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:32.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:33.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:33.126+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:34.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:34.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:35.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:35.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:36.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:36.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:36.968 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:36.998 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:36.999 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:51:36.999 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:36.999 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:51:37.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:37.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:37.996 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2116,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:38.018 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:38.018 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:51:38.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:38.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:39.027 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2116,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:39.057 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:39.058 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:51:39.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:39.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:40.050 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2116,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:40.071 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:51:40.071 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:40.071 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:51:40.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:40.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:41.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:51:41.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:41.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:42.176 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:51:42.177 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:42.177 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:51:42.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:42.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:43.196 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:43.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:43.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:43.222 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2116,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:51:43.223 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:51:43.223 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:51:43.223 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:43.223 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:51:44.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:44.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:44.296 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2116,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:44.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:44.329 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:51:45.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:45.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:45.329 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2116,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:45.354 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:51:45.354 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:51:46.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:46.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:46.379 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2116,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:51:46.397 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:51:46.397 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T10:51:46.398 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:51:46.398 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:51:46.398 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:51:46.398 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:51:46.418 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:51:46.418 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:51:46.418 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:51:46.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:51:47.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:47.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:48.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:48.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:49.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:49.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:50.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:50.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:51.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:51.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:52.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:52.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:53.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:53.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:54.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:54.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:55.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:55.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:56.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:56.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:57.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:57.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:58.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:58.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:51:59.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:51:59.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:00.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:00.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:01.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:01.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:01.419 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:52:01.420 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:52:01.420 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:52:01.420 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:52:01.420 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:52:02.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:02.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:03.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:03.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:04.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:04.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:05.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:05.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:06.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:06.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:07.091 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2118,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:52:07.113 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2118,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:52:07.113 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:52:07.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:52:07.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:52:07.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:07.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:08.131 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2118,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:52:08.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:52:08.154 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:52:08.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:08.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:09.144 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2118,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:52:09.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:52:09.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:09.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:10.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:10.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:11.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:11.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:12.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:12.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:13.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:13.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:14.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:14.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:15.274 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:15.272+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:16.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:16.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:17.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:17.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:18.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:18.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:19.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:19.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:20.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:20.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:21.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:21.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:22.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:22.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:23.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:23.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:24.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:24.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:25.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:25.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:26.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:26.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:27.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:27.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:28.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:28.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:29.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:29.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:30.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:30.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:31.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:31.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:32.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:32.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:33.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:33.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:34.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:34.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:35.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:35.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:36.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:36.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:37.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:37.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:38.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:38.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:39.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:39.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:40.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:40.321+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:41.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:41.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:42.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:42.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:43.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:43.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:44.285 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:44.283+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:45.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:45.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:46.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:46.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:47.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:47.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:48.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:48.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:49.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:49.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:50.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:50.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:51.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:51.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:52.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:52.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:53.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:53.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:54.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:54.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:55.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:55.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:56.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:56.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:57.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:57.345+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:58.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:58.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:52:59.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:52:59.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:00.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:00.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:01.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:01.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:02.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:02.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:03.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:03.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:04.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:04.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:05.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:05.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:06.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:06.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:07.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:07.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:08.400 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:08.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:09.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:09.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:10.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:10.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:11.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:11.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:12.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:12.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:13.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:13.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:14.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:14.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:15.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:15.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:16.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:16.428+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:17.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:17.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:18.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:18.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:19.369 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:19.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:20.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:20.352+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:21.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:21.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:22.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:22.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:23.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:23.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:24.371 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:24.369+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:25.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:25.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:26.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:26.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:27.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:27.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:28.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:28.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:29.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:29.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:30.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:30.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:31.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:31.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:32.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:32.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:33.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:33.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:34.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:34.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:35.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:35.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:36.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:36.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:37.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:37.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:38.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:38.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:39.174 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:53:39.175 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:53:39.175 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:53:39.176 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:39.176 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:53:39.184 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:53:39.184 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T10:53:39.185 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T10:53:39.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T10:53:39.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:39.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:40.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:40.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:41.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:41.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:42.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:42.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:43.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:43.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:44.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:44.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:45.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:45.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:46.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:46.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:47.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:47.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:48.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:48.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:49.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:49.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:50.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:50.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:51.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:51.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:52.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:52.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:53.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:53.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:54.187 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T10:53:54.187 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T10:53:54.187 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:53:54.187 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:54.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:53:54.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:54.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:55.165 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:53:55.194 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:53:55.194 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:53:55.194 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:55.195 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:53:55.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:55.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:56.172 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2122,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:53:56.196 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:56.197 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:53:56.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:56.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:57.209 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2122,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:53:57.240 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:57.240 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:53:57.438 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:57.436+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:58.236 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2122,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:53:58.266 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:53:58.266 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:53:58.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:53:58.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:58.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:53:59.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:53:59.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:53:59.423+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:00.384 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:54:00.384 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:00.384 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:54:00.413 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:00.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:01.400 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:01.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:01.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:01.431 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:01.431 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:54:01.431 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:54:01.431 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:01.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:54:02.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:02.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:02.405 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2122,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:54:02.428 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:02.428 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:54:03.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:03.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:03.393 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2122,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:54:03.417 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:03.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:54:04.392 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2122,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:54:04.404 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:04.403+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:04.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T10:54:04.415 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:54:04.415 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T10:54:04.416 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:54:04.416 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T10:54:04.416 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:54:04.435 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T10:54:04.435 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:54:04.435 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:04.435 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:54:05.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:05.379+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:05.457 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:05.487 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:06.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:06.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:07.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:07.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:08.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:08.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:08.488 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:08.489 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:54:09.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:09.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:09.461 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:09.484 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:10.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:10.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:11.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:11.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:12.431 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:12.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:12.485 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:12.486 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:54:13.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:13.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:13.482 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:13.512 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2122,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:14.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:14.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:15.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:15.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:16.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:16.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:16.513 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:16.513 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:54:17.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:17.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:18.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:18.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:19.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:19.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:19.924 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2126,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:19.955 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2126,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:54:19.955 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:54:19.956 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:19.956 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:54:20.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:20.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:20.965 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2126,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:54:20.997 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:54:20.997 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:54:21.367 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:21.365+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:21.999 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2126,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:54:22.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:54:22.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:22.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:23.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:23.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:24.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:24.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:25.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:25.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:26.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:26.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:27.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:27.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:28.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:28.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:29.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:29.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:30.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:30.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:31.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:31.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:32.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:32.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:33.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:33.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:34.383 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:34.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:35.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:35.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:36.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:36.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:37.392 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:37.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:38.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:38.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:39.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:39.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:40.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:40.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:41.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:41.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:42.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:42.464+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:43.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:43.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:44.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:44.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:45.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:45.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:46.579 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:46.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:47.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:47.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:48.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:48.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:49.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:49.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:50.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:50.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:51.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:51.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:52.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:52.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:53.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:53.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:54.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:54.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:55.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:55.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:56.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:56.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:57.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:57.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:58.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:58.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:54:59.418 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:54:59.416+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:00.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:00.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:01.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:01.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:02.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:02.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:03.497 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:03.495+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:04.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:04.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:05.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:05.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:06.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:06.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:07.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:07.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:08.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:08.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:09.420 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:09.417+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:10.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:10.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:11.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:11.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:12.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:12.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:13.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:13.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:14.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:14.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:15.409 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:15.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:16.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:16.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:17.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:17.437+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:18.394 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:18.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:19.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:19.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:20.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:20.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:21.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:21.305+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:22.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:22.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:23.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:23.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:24.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:24.367+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:25.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:25.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:26.355 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:26.353+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:27.326 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:27.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:28.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:28.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:29.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:29.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:30.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:30.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:31.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:31.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:32.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:32.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:33.381 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:33.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:34.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:34.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:35.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:35.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:36.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:36.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:37.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:37.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:38.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:38.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:39.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:39.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:40.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:40.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:41.425 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:41.424+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:42.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:42.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:43.419 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:43.418+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:44.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:44.388+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:45.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:45.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:46.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:46.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:47.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:47.377+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:48.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:48.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:49.340 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:49.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:50.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:50.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:51.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:51.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:52.022 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T10:55:52.023 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T10:55:52.023 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T10:55:52.023 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:52.023 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T10:55:52.035 INFO:tasks.ceph.mon.a:Started 2019-10-01T10:55:52.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:55:52.035 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:52.035 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:55:52.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:52.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:53.017 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2126,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:55:53.039 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2126,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:55:53.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:53.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:54.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:54.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:55.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:55.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:56.040 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:56.040 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:55:56.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:56.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:57.032 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:55:57.054 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:55:57.055 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:55:57.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:57.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:55:57.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:57.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:58.065 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2128,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:55:58.087 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:58.088 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:55:58.323 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:58.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:55:59.084 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2128,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:55:59.106 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:55:59.107 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:55:59.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:55:59.297+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:00.105 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2128,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:00.128 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:56:00.128 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:00.128 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:56:00.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:00.326+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:01.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:56:01.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:01.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:02.156 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:56:02.156 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:02.156 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:56:02.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:02.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:03.151 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:03.173 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:03.173 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:56:03.173 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:56:03.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:03.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:56:03.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:03.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:04.175 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2128,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:04.207 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:04.207 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:56:04.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:04.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:05.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2128,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:05.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:05.234 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:56:05.288 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:05.286+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:06.212 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2128,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:06.233 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T10:56:06.233 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:56:06.233 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T10:56:06.233 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:56:06.234 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T10:56:06.234 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:56:06.257 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T10:56:06.258 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:56:06.258 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:06.258 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:56:06.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:06.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:07.269 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:07.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:07.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:07.301 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2128,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:08.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:08.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:09.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:09.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:10.302 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:10.303 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:56:10.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:10.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:11.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:11.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:12.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:12.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:13.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:13.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:14.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:14.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:15.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:15.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:16.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:16.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:17.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:17.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:18.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:18.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:19.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:19.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:20.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:20.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:21.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:21.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:21.271 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:21.302 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:56:21.303 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:56:21.303 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:21.303 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:56:22.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:22.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:22.370 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2130,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:22.400 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:56:22.400 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:56:23.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:23.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:23.413 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2130,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:56:23.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:56:24.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:24.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:25.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:25.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:26.031 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:26.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:27.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:27.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:28.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:28.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:29.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:29.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:30.129 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:30.127+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:31.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:31.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:32.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:32.058+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:33.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:33.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:34.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:34.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:35.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:35.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:36.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:36.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:37.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:37.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:38.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:38.191+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:39.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:39.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:40.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:40.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:41.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:41.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:42.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:42.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:43.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:43.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:44.123 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:44.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:45.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:45.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:46.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:46.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:47.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:47.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:48.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:48.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:49.130 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:49.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:50.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:50.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:51.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:51.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:52.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:52.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:53.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:53.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:54.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:54.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:55.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:55.107+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:56.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:56.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:57.176 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:57.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:58.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:58.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:56:59.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:56:59.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:00.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:00.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:01.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:01.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:02.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:02.177+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:03.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:03.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:04.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:04.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:05.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:05.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:06.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:06.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:07.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:07.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:08.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:08.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:09.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:09.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:10.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:10.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:11.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:11.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:12.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:12.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:13.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:13.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:14.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:14.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:15.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:15.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:16.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:16.122+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:17.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:17.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:18.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:18.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:19.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:19.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:20.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:20.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:21.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:21.156+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:22.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:22.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:23.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:23.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:24.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:24.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:25.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:25.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:26.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:26.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:27.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:27.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:28.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:28.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:29.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:29.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:30.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:30.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:31.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:31.073+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:32.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:32.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:33.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:33.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:34.045 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:34.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:35.008 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:35.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:36.002 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:36.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:37.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:37.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:38.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:38.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:39.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:39.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:40.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:40.080+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:41.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:41.094+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:42.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:42.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:43.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:43.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:44.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:44.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:45.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:45.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:46.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:46.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:47.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:47.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:48.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:48.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:49.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:49.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:50.072 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:50.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:51.113 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:51.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:52.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:52.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:53.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:53.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:53.438 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T10:57:53.438 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T10:57:53.438 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T10:57:53.439 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T10:57:53.439 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T10:57:53.445 INFO:tasks.ceph.mon.b:Started 2019-10-01T10:57:53.445 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:57:53.445 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:57:53.445 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:57:54.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:54.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:54.450 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:57:54.473 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:57:55.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:55.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:56.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:56.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:57.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:57.102+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:57.474 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:57:57.474 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:57:58.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:58.061+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:57:58.458 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:57:58.490 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:57:59.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:57:59.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:00.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:00.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:01.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:01.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:01.492 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:01.492 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:02.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:02.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:02.512 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:02.540 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:03.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:03.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:04.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:04.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:05.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:05.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:05.541 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:05.542 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:06.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:06.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:06.558 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:06.588 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2130,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:07.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:07.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:08.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:08.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:09.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:09.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:09.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:09.589 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:10.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:10.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:11.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:11.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:12.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:12.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:13.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:13.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:14.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:14.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:15.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:15.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:16.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:16.098+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:17.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:17.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:18.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:18.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:19.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:19.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:20.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:20.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:20.602 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:20.634 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:20.634 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:58:20.635 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:20.635 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:58:20.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:20.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:21.637 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2144,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:21.668 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:21.669 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:58:21.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:21.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:22.621 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2144,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:22.643 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:22.644 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:58:23.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:23.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:23.614 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2144,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:23.645 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T10:58:23.645 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:23.645 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T10:58:24.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:24.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:24.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T10:58:25.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:25.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:25.681 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T10:58:25.682 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:25.682 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:26.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:26.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:26.625 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:26.655 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:26.655 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T10:58:26.656 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T10:58:26.656 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:26.656 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:58:26.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:26.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:27.559 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2144,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:27.588 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:27.588 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T10:58:27.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:27.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:28.566 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2144,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:28.589 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:28.589 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:58:29.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:29.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:29.589 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2144,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:29.612 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T10:58:29.612 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T10:58:29.612 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T10:58:29.612 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T10:58:29.612 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T10:58:29.613 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T10:58:29.628 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T10:58:29.628 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T10:58:29.629 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:29.629 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:29.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:29.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:30.682 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:30.714 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:30.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:30.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:31.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:31.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:32.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:32.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:33.715 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:33.715 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:33.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:33.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:34.714 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:34.737 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:34.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:34.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:35.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:35.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:36.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:36.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:37.737 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:37.737 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:37.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:37.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:38.730 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:38.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:38.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:38.760 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2144,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:39.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:39.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:40.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:40.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:41.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:41.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:41.761 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:41.761 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T10:58:42.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:42.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:43.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:43.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:44.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:44.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:45.235 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2148,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:45.265 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2148,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T10:58:45.265 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T10:58:45.265 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:45.266 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T10:58:45.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:45.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:46.268 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2148,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:46.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T10:58:46.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T10:58:46.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:46.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:47.302 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2148,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T10:58:47.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T10:58:47.738 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:47.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:48.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:48.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:49.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:49.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:50.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:50.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:51.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:51.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:52.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:52.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:53.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:53.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:54.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:54.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:55.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:55.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:56.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:56.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:57.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:57.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:58.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:58.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:58:59.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:58:59.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:00.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:00.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:01.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:01.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:02.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:02.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:03.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:03.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:04.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:04.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:05.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:05.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:06.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:06.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:07.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:07.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:08.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:08.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:09.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:09.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:10.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:10.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:11.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:11.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:12.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:12.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:13.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:13.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:14.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:14.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:15.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:15.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:16.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:16.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:17.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:17.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:18.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:18.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:19.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:19.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:20.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:20.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:21.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:21.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:22.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:22.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:23.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:23.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:24.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:24.812+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:25.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:25.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:26.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:26.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:27.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:27.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:28.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:28.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:29.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:29.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:30.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:30.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:31.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:31.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:32.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:32.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:33.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:33.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:34.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:34.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:35.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:35.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:36.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:36.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:37.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:37.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:39.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:38.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:40.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:40.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:41.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:41.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:42.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:42.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:43.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:43.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:44.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:44.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:45.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:45.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:46.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:46.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:47.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:47.184+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:48.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:48.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:49.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:49.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:50.180 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:50.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:51.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:51.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:52.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:52.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:53.102 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:53.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:54.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:54.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:55.092 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:55.090+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:56.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:56.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:57.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:57.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:58.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:58.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T10:59:59.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T10:59:59.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:00.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:00.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:01.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:01.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:02.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:02.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:03.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:03.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:04.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:04.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:05.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:05.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:06.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:06.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:07.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:07.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:08.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:08.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:09.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:08.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:09.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:09.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:11.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:11.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:11.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:11.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:13.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:13.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:14.034 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:14.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:15.060 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:15.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:16.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:16.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:17.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:17.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:17.333 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:00:17.333 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:00:17.333 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:00:17.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:17.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:00:17.340 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:00:17.340 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:00:17.341 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:17.341 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:18.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:18.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:18.313 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2148,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:18.335 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2148,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:19.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:19.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:20.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:20.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:21.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:21.086+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:21.336 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:21.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:22.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:22.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:23.089 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:23.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:24.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:24.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:25.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:25.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:26.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:26.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:26.231 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:26.253 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:26.253 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:00:26.253 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:26.253 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:00:27.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:27.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:27.192 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2154,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:27.222 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:27.222 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:00:28.038 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T11:00:28.036+0000 7f607b7fe700 0 --2- 172.21.6.124:0/610904241 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f607c07b210 0x7f607c074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T11:00:28.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:28.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:28.663 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2154,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:28.685 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:28.685 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:00:29.146 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:29.144+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:29.694 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2154,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:29.716 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:00:29.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:29.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:00:30.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:30.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:30.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:00:31.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:31.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:31.839 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:00:31.839 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:31.839 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:32.140 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:32.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:32.859 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:32.888 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:32.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:00:32.889 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:00:32.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:32.889 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:00:33.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:33.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:33.879 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2154,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:33.902 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:33.903 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:00:34.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:34.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:34.892 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2154,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:34.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:34.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:00:35.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:35.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:35.897 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2154,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:35.919 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:00:35.920 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:00:35.920 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:00:35.920 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:00:35.920 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:00:35.920 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:00:35.943 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:00:35.943 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:00:35.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:35.943 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:36.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:36.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:36.945 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:36.963 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:37.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:37.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:38.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:38.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:39.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:39.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:39.964 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:39.964 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:40.233 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:40.231+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:40.966 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:40.989 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2154,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:41.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:41.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:42.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:42.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:43.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:43.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:43.990 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:43.990 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:00:44.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:44.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:45.264 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:45.262+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:46.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:46.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:47.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:47.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:48.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:48.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:49.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:49.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:50.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:50.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:51.104 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:51.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:51.508 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:51.529 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:00:51.529 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:00:51.529 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:51.529 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:00:52.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:52.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:52.501 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2156,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:00:52.531 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:00:52.531 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:00:53.025 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:53.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:53.567 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2156,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:00:53.593 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:00:54.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:54.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:55.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:54.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:56.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:56.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:57.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:57.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:58.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:58.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:00:59.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:00:59.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:00.054 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:00.052+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:01.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:01.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:02.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:02.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:02.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:02.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:03.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:03.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:04.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:04.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:05.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:05.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:06.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:06.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:07.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:07.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:08.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:08.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:09.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:09.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:10.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:10.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:11.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:11.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:12.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:12.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:13.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:13.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:14.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:14.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:15.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:15.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:16.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:16.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:17.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:17.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:18.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:18.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:19.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:19.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:20.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:20.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:21.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:21.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:22.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:22.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:23.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:23.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:24.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:24.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:25.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:25.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:26.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:26.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:27.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:27.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:28.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:28.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:29.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:29.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:30.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:30.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:31.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:31.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:32.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:32.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:33.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:33.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:34.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:34.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:35.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:35.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:36.814 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:36.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:37.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:37.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:38.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:38.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:39.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:39.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:40.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:40.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:41.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:41.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:42.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:42.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:43.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:43.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:44.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:44.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:46.026 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:46.024+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:47.044 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:47.043+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:48.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:48.020+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:49.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:49.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:50.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:50.029+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:50.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:50.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:52.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:52.021+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:53.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:53.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:54.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:54.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:55.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:55.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:56.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:56.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:57.057 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:57.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:58.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:58.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:01:59.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:01:59.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:00.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:00.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:01.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:01.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:02.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:02.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:03.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:03.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:04.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:04.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:05.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:05.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:06.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:06.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:07.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:07.118+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:08.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:08.085+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:09.076 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:09.075+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:10.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:10.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:11.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:11.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:12.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:12.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:13.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:13.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:14.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:14.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:15.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:15.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:16.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:16.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:17.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:17.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:18.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:18.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:19.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:19.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:20.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:20.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:21.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:21.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:22.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:22.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:23.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:23.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:23.594 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:02:23.595 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:02:23.595 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:02:23.595 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:02:23.595 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:02:23.603 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:02:23.603 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:02:23.603 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:23.603 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:24.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:24.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:24.628 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:24.652 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:25.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:25.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:26.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:26.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:27.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:27.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:27.657 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:27.657 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:28.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:28.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:28.652 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:28.684 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2156,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:29.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:29.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:30.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:30.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:31.368 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:31.366+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:31.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:31.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:32.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:32.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:33.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:33.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:34.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:34.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:35.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:35.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:36.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:36.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:37.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:37.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:37.542 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:37.565 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:37.565 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:02:37.565 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:37.566 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:02:38.287 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:38.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:38.549 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2166,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:38.568 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:38.568 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:02:39.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:39.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:39.506 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2166,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:39.533 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:39.534 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:02:40.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:40.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:40.438 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2166,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:40.461 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:02:40.462 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:40.462 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:02:41.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:41.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:41.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:02:42.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:42.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:42.592 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:02:42.592 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:42.592 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:43.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:43.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:43.642 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:43.663 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:43.663 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:02:43.663 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:02:43.664 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:43.664 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:02:44.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:44.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:44.587 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2166,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:44.616 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:44.616 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:02:45.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:45.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:45.630 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2166,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:45.653 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:45.653 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:02:46.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:46.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:46.636 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2166,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:02:46.666 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:02:46.666 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:02:46.667 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:02:46.667 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:02:46.667 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:02:46.667 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:02:46.686 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:02:46.686 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:02:46.686 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:46.686 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:47.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:47.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:47.643 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:47.664 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:48.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:48.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:49.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:49.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:50.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:50.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:50.666 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:50.666 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:51.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:51.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:51.665 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:51.697 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:52.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:52.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:53.247 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:53.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:54.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:54.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:54.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:54.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:55.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:55.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:55.782 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:55.813 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2166,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:02:56.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:56.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:57.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:57.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:58.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:58.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:02:58.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:02:58.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:02:59.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:02:59.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:00.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:00.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:01.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:01.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:02.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:02.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:02.857 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:03:02.879 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:03:02.879 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:03:02.879 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:03:02.879 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:03:03.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:03.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:03.851 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2168,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:03:03.877 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:03:03.877 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:03:04.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:04.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:04.860 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2168,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:03:04.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:03:05.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:05.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:06.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:06.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:07.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:07.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:08.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:08.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:09.229 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:09.227+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:10.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:10.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:11.190 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:11.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:12.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:12.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:13.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:13.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:14.210 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:14.208+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:15.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:15.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:16.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:16.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:17.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:17.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:18.196 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:18.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:19.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:19.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:20.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:20.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:21.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:21.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:22.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:22.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:23.283 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:23.281+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:24.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:24.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:25.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:25.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:26.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:26.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:27.375 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:27.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:28.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:28.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:29.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:29.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:30.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:30.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:31.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:31.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:32.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:32.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:33.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:33.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:34.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:34.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:35.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:35.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:36.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:36.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:37.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:37.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:38.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:38.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:39.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:39.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:40.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:40.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:41.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:41.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:42.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:42.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:43.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:43.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:44.448 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:44.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:45.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:45.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:46.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:46.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:47.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:47.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:48.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:48.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:49.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:49.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:50.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:50.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:51.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:51.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:52.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:52.406+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:53.421 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:53.419+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:54.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:54.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:55.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:55.445+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:56.430 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:56.429+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:57.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:57.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:58.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:58.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:03:59.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:03:59.398+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:00.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:00.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:01.399 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:01.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:02.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:02.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:03.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:03.373+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:04.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:04.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:05.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:05.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:06.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:06.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:07.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:07.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:08.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:08.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:09.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:09.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:10.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:10.435+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:11.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:11.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:12.428 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:12.426+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:13.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:13.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:14.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:14.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:15.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:15.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:16.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:16.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:17.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:17.407+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:18.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:18.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:19.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:19.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:20.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:20.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:21.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:21.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:22.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:22.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:23.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:23.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:24.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:24.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:25.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:25.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:26.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:26.312+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:27.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:27.309+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:28.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:28.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:29.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:29.310+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:30.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:30.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:31.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:31.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:32.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:32.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:33.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:33.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:34.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:34.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:34.897 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:04:34.897 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:04:34.897 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:04:34.897 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:04:34.897 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:04:34.903 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:04:34.903 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:04:34.903 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:34.904 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:35.397 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:35.395+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:35.891 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:35.914 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:36.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:36.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:37.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:37.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:38.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:38.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:38.915 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:38.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:39.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:39.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:39.918 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:39.948 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2168,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:40.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:40.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:41.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:41.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:42.379 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:42.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:42.949 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:42.949 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:43.332 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:43.330+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:43.964 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:43.986 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:43.986 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:04:43.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:43.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:04:44.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:44.336+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:44.989 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2172,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:45.020 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:45.020 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:04:45.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:45.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:46.161 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2172,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:46.182 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:46.182 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:04:46.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:46.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:47.135 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2172,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:47.160 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:04:47.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:47.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:04:47.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:47.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:48.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:48.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:48.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:04:49.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:49.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:49.290 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:04:49.291 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:49.291 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:50.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:50.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:50.303 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:50.327 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":8,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:50.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:04:50.327 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:04:50.327 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:50.328 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:04:51.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:51.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:51.326 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2172,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:51.351 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:51.351 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:04:52.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:52.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:52.310 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2172,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:52.334 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:52.335 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:04:53.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:53.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:53.276 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2172,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:04:53.306 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:04:53.306 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:04:53.306 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:04:53.307 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:04:53.307 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:04:53.307 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:04:53.326 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:04:53.326 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:04:53.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:53.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:54.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:54.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:54.312 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:54.343 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:55.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:55.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:56.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:56.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:57.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:57.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:57.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:04:57.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:04:58.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:58.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:04:58.359 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:58.377 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:04:59.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:04:59.182+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:00.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:00.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:01.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:01.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:01.378 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:05:01.379 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:05:02.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:02.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:02.381 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:05:02.404 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2172,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":20,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:05:03.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:03.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:04.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:04.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:05.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:05.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:05.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:05:05.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:05:06.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:06.128+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:07.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:07.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:08.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:08.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:08.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:05:08.290 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:05:08.290 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:05:08.290 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:05:08.291 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:05:09.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:09.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:09.247 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2174,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:05:09.278 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:05:09.278 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:05:10.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:10.055+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:10.211 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2174,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:05:10.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:05:11.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:11.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:12.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:12.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:13.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:13.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:14.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:14.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:14.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:14.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:15.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:15.990+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:16.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:16.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:18.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:18.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:19.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:19.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:20.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:20.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:21.004 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:21.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:22.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:22.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:23.088 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:23.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:24.114 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:24.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:25.153 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:25.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:26.131 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:26.129+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:27.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:27.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:28.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:28.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:29.125 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:29.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:30.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:30.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:31.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:31.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:32.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:32.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:33.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:33.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:34.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:34.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:35.166 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:35.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:36.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:36.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:37.127 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:37.125+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:38.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:38.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:39.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:39.164+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:40.138 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:40.136+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:41.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:41.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:42.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:42.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:43.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:43.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:44.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:44.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:45.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:45.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:46.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:46.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:47.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:47.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:48.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:48.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:49.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:49.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:50.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:50.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:51.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:51.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:52.106 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:52.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:53.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:53.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:54.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:54.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:55.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:55.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:56.142 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:56.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:57.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:57.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:58.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:58.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:05:59.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:05:59.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:00.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:00.112+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:01.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:01.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:02.132 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:02.130+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:03.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:03.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:04.094 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:04.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:05.137 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:05.135+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:06.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:06.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:07.083 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:07.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:08.055 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:08.053+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:09.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:09.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:10.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:10.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:11.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:11.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:12.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:12.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:13.081 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:13.079+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:14.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:14.074+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:15.074 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:15.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:16.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:16.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:16.985 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:16.983+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:17.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:17.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:18.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:18.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:19.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:19.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:20.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:20.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:21.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:21.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:22.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:22.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:23.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:23.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:24.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:24.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:25.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:25.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:26.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:26.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:27.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:27.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:28.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:28.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:29.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:29.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:30.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:30.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:31.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:31.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:32.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:32.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:33.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:33.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:34.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:34.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:35.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:35.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:37.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:36.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:38.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:38.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:39.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:39.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:40.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:40.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:40.242 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:06:40.243 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:06:40.243 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:06:40.243 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:06:40.243 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:06:40.251 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:06:40.252 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:06:40.252 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:40.252 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:06:40.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:40.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:41.239 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:41.269 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:42.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:42.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:42.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:42.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:44.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:44.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:44.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:44.270 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:06:45.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:44.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:45.273 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:45.297 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2174,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:46.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:45.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:47.018 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:47.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:47.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:47.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:48.299 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:48.299 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:06:48.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:48.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:49.313 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:49.343 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:49.343 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:06:49.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:49.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:06:49.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:49.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:50.317 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2178,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:50.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:50.337 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:06:51.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:51.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:51.218 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2178,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:51.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:51.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:06:52.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:52.050+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:52.205 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2178,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:52.227 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:06:52.227 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:52.227 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:06:53.070 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:53.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:53.272 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:06:54.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:54.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:54.273 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:06:54.273 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:54.273 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:06:55.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:55.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:55.243 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:55.264 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:55.264 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:06:55.264 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:06:55.264 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:55.265 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:06:56.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:56.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:56.246 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2178,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:56.286 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:56.286 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:06:57.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:57.111+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:57.276 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2178,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:57.307 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:57.307 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:06:58.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:58.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:58.325 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2178,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:06:58.355 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:06:58.355 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:06:58.355 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:06:58.355 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:06:58.356 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:06:58.356 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:06:58.373 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:06:58.374 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:06:58.374 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:06:58.374 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:06:59.120 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:06:59.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:06:59.367 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:06:59.397 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:00.103 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:00.101+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:01.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:01.088+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:02.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:02.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:02.398 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:07:02.399 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:07:03.159 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:03.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:03.401 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:03.424 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:04.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:04.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:05.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:05.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:06.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:06.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:06.425 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:07:06.425 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:07:07.177 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:07.175+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:07.407 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:07.430 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2178,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:08.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:08.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:09.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:09.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:10.197 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:10.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:10.431 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:07:10.431 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:07:11.238 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:11.236+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:12.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:12.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:13.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:13.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:14.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:14.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:14.465 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:14.484 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:07:14.484 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:07:14.485 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:07:14.485 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:07:15.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:15.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:15.482 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2180,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:07:15.508 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:07:15.508 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:07:16.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:16.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:16.510 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2180,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:07:16.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:07:17.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:17.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:18.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:18.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:19.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:19.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:20.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:20.200+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:21.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:21.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:22.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:22.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:23.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:23.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:24.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:24.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:25.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:25.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:26.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:26.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:27.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:27.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:28.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:28.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:29.243 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:29.241+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:30.259 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:30.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:31.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:31.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:32.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:32.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:33.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:33.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:34.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:34.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:35.192 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:35.190+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:36.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:36.158+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:37.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:37.178+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:38.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:38.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:39.156 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:39.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:40.110 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:40.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:41.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:41.099+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:42.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:42.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:43.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:43.121+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:44.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:44.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:45.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:45.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:46.122 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:46.120+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:47.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:47.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:48.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:48.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:49.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:49.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:50.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:50.225+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:51.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:51.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:52.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:52.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:53.270 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:53.269+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:54.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:54.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:55.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:55.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:56.255 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:56.253+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:57.236 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:57.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:58.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:58.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:07:59.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:07:59.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:00.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:00.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:01.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:01.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:02.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:02.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:03.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:03.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:04.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:04.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:05.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:05.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:06.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:06.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:07.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:07.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:08.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:08.285+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:09.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:09.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:10.337 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:10.335+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:11.373 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:11.371+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:12.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:12.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:13.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:13.397+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:14.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:14.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:15.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:15.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:16.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:16.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:17.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:17.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:18.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:18.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:19.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:19.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:20.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:20.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:21.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:21.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:22.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:22.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:23.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:23.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:24.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:24.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:25.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:25.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:26.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:26.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:27.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:27.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:28.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:28.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:29.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:29.682+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:30.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:30.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:31.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:31.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:32.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:32.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:33.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:33.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:34.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:34.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:35.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:35.704+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:36.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:36.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:37.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:37.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:38.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:38.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:39.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:39.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:40.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:40.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:41.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:41.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:42.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:42.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:43.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:43.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:44.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:44.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:45.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:45.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:46.542 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:08:46.543 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:08:46.543 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:08:46.543 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:08:46.543 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:08:46.550 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:08:46.550 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:08:46.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:08:46.550 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:08:46.660 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:46.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:47.539 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:47.569 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:47.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:47.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:48.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:48.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:49.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:49.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:50.570 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:08:50.571 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:08:50.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:50.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:51.580 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:51.602 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:51.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:51.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:52.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:52.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:53.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:53.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:54.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:08:54.605 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:08:54.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:54.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:55.622 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:55.642 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":102,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:55.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:55.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:56.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:56.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:57.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:57.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:58.649 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:08:58.649 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:08:58.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:58.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:08:59.710 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:59.733 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2180,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":106,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:08:59.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:08:59.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:00.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:00.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:01.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:01.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:02.734 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:02.734 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:02.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:02.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:03.722 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:03.754 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:03.754 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:09:03.754 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:03.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:09:03.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:03.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:04.780 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2184,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:04.802 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:04.802 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:09:04.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:04.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:05.780 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2184,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:05.812 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:05.812 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:09:05.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:05.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:06.811 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2184,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:06.833 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:09:06.834 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:06.834 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:09:06.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:06.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:07.875 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:09:07.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:07.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:08.876 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:09:08.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:08.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:08.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:08.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:09.872 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:09.901 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:09.902 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:09:09.902 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:09:09.902 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:09.902 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:09:09.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:09.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:10.910 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2184,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:10.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:10.943 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:09:10.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:10.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:11.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:11.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:11.963 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2184,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:11.985 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:11.985 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:09:12.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:12.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:13.046 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2184,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:13.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:09:13.068 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:09:13.068 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:09:13.069 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:09:13.069 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:09:13.069 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:09:13.086 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:09:13.086 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:09:13.086 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:13.087 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:13.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:13.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:14.149 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:14.172 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:14.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:14.968+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:15.994 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:15.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:16.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:16.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:17.174 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:17.174 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:17.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:17.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:18.183 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:18.213 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:19.028 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:19.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:20.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:20.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:21.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:21.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:21.214 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:21.215 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:22.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:22.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:22.225 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:22.247 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2184,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:23.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:23.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:24.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:24.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:25.073 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:25.070+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:25.249 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:25.249 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:09:26.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:26.069+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:27.077 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:27.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:28.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:28.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:29.101 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:29.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:29.300 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2188,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:29.332 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2188,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:09:29.332 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:09:29.333 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:29.333 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:09:30.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:30.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:30.346 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2188,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:09:30.377 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:09:30.377 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:09:31.053 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:31.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:31.360 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2188,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:09:31.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:09:32.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:32.032+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:33.027 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:33.026+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:34.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:34.012+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:35.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:35.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:35.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:35.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:36.989 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:36.987+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:37.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:37.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:39.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:39.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:39.988 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:39.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:40.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:40.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:41.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:41.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:42.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:42.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:43.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:43.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:44.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:44.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:45.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:45.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:46.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:46.986+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:48.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:48.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:49.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:49.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:50.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:50.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:50.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:50.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:51.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:51.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:52.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:52.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:53.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:53.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:54.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:54.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:55.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:55.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:56.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:56.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:57.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:57.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:58.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:58.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:09:59.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:09:59.779+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:00.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:00.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:01.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:01.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:02.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:02.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:03.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:03.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:04.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:04.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:05.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:05.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:06.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:06.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:07.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:07.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:08.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:08.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:09.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:09.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:10.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:10.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:11.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:11.754+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:12.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:12.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:13.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:13.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:14.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:14.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:15.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:15.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:16.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:16.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:17.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:17.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:18.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:18.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:19.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:19.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:20.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:20.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:21.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:21.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:22.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:22.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:23.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:23.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:24.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:24.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:25.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:25.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:26.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:26.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:27.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:27.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:28.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:28.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:29.786 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:29.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:30.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:30.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:31.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:31.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:32.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:32.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:33.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:33.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:34.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:34.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:35.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:35.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:36.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:36.771+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:37.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:37.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:38.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:38.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:39.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:39.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:40.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:40.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:41.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:41.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:42.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:42.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:43.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:43.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:44.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:44.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:45.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:45.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:46.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:46.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:47.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:47.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:48.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:48.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:49.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:49.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:50.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:50.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:51.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:51.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:52.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:52.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:53.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:53.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:54.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:54.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:55.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:55.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:56.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:56.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:57.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:57.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:58.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:58.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:10:59.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:10:59.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:00.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:00.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:01.391 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:11:01.391 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:11:01.392 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:11:01.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:01.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:11:01.399 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:11:01.399 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:11:01.399 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:01.400 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:01.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:01.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:02.419 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2188,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:02.449 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2188,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:02.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:02.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:03.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:03.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:04.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:04.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:05.451 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:05.451 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:05.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:05.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:06.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:06.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:07.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:07.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:08.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:08.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:09.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:09.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:10.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:10.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:11.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:11.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:12.657 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:12.679 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:12.679 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:11:12.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:12.680 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:11:12.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:12.727+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:13.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2194,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:11:13.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:13.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:11:13.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:13.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:14.667 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2194,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:11:14.688 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:14.688 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:11:14.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:14.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:15.686 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2194,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:11:15.717 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:11:15.717 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:15.717 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:11:15.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:15.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:16.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:16.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:16.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:11:17.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:17.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:17.850 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:11:17.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:17.851 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:18.854 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:18.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:18.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:18.884 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:18.885 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:11:18.885 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:11:18.885 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:18.885 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:11:19.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:19.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:19.906 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2194,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:11:19.933 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:19.934 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:11:20.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:20.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:20.923 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2194,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:11:20.946 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:20.946 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:11:21.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:21.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:21.935 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2194,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:11:21.967 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:11:21.967 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:11:21.967 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:11:21.967 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:11:21.967 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:11:21.968 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:11:22.002 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:11:22.002 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:11:22.003 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:22.003 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:22.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:22.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:22.964 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:22.985 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:23.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:23.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:24.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:24.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:25.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:25.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:25.986 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:25.986 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:26.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:26.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:26.985 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:27.015 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:27.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:27.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:28.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:28.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:29.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:29.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:30.015 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:30.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:30.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:30.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:31.027 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:31.049 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2194,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:31.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:31.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:32.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:32.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:33.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:33.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:34.050 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:34.050 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:11:34.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:34.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:35.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:35.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:36.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:36.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:37.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:37.937+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:38.081 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:38.114 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:11:38.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:11:38.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:38.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:11:38.955 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:38.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:39.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2196,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:11:39.142 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:11:39.142 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:11:39.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:39.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:40.149 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2196,"quorum":[1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:11:40.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:11:40.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:40.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:41.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:41.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:42.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:42.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:43.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:43.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:44.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:44.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:45.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:45.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:46.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:46.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:47.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:47.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:48.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:48.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:49.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:49.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:50.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:50.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:51.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:51.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:52.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:52.768+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:53.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:53.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:54.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:54.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:55.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:55.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:56.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:56.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:57.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:57.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:58.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:58.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:11:59.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:11:59.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:00.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:00.792+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:01.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:01.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:02.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:02.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:03.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:03.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:04.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:04.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:05.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:05.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:06.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:06.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:07.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:07.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:08.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:08.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:09.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:09.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:10.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:10.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:11.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:11.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:12.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:12.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:13.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:13.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:14.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:14.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:15.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:15.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:16.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:16.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:17.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:17.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:18.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:18.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:19.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:19.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:20.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:20.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:21.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:21.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:22.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:22.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:23.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:23.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:24.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:24.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:25.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:25.907+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:26.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:26.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:27.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:27.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:28.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:28.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:29.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:29.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:30.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:30.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:31.923 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:31.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:32.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:32.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:33.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:33.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:34.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:34.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:35.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:35.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:36.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:36.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:37.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:37.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:38.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:38.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:39.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:39.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:40.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:40.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:41.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:41.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:42.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:42.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:43.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:43.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:44.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:44.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:45.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:45.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:46.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:46.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:47.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:47.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:48.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:48.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:49.843 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:49.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:50.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:50.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:51.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:51.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:52.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:52.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:53.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:53.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:54.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:54.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:55.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:55.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:56.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:56.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:57.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:57.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:58.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:58.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:12:59.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:12:59.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:00.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:00.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:01.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:01.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:02.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:02.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:03.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:03.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:04.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:04.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:05.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:05.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:06.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:06.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:07.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:07.954+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:08.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:08.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:09.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:09.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:10.182 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:13:10.182 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:13:10.182 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:13:10.182 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:13:10.182 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:13:10.188 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:13:10.189 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:13:10.189 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:10.189 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:10.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:10.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:11.179 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:11.203 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:11.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:11.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:12.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:12.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:13.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:13.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:14.204 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:14.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:14.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:14.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:15.202 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:15.233 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2196,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:15.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:15.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:16.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:16.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:17.858 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:17.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:18.234 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:18.235 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:18.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:18.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:19.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:19.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:20.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:20.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:21.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:21.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:22.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:22.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:23.772 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:23.803 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:23.803 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:13:23.803 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:23.803 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:13:23.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:23.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:24.804 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2204,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:13:24.829 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:24.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:13:24.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:24.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:25.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:25.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:25.881 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2204,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:13:25.914 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:25.915 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:13:26.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:26.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:26.992 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2204,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:13:27.014 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:13:27.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:27.015 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:13:27.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:27.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:28.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:13:28.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:28.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:29.062 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:13:29.063 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:29.063 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:29.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:29.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:30.016 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:30.037 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:30.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:13:30.038 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:13:30.038 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:30.038 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:13:30.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:30.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:30.950 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2204,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:13:30.971 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:30.971 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:13:31.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:31.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:31.952 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2204,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:13:31.982 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:31.983 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:13:32.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:32.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:32.991 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2204,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:13:33.013 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:13:33.013 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:13:33.013 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:13:33.013 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:13:33.014 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:13:33.014 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:13:33.031 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:13:33.031 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:13:33.032 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:33.032 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:33.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:33.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:34.079 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:34.101 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:34.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:34.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:35.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:35.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:36.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:36.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:37.102 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:37.102 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:37.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:37.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:38.121 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:38.144 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:38.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:38.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:39.779 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:39.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:40.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:40.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:41.145 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:41.146 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:41.781 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:41.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:42.127 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:42.159 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2204,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:42.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:42.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:43.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:43.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:44.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:44.805+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:45.160 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:45.160 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:13:45.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:45.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:46.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:46.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:47.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:47.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:48.250 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2208,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:48.282 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2208,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:13:48.282 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:13:48.282 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:48.282 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:13:48.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:48.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:49.303 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2208,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:13:49.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:13:49.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:13:49.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:49.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:50.366 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2208,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:13:50.396 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:13:50.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:50.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:51.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:51.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:52.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:52.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:53.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:53.840+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:54.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:54.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:55.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:55.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:56.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:56.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:57.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:57.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:58.811 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:58.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:13:59.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:13:59.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:00.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:00.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:01.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:01.817+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:02.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:02.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:03.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:03.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:04.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:04.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:05.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:05.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:06.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:06.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:07.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:07.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:08.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:08.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:09.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:09.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:10.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:10.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:11.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:11.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:12.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:12.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:13.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:13.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:14.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:14.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:15.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:15.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:16.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:16.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:18.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:18.000+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:18.962 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:18.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:19.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:19.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:20.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:20.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:21.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:21.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:22.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:22.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:23.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:23.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:24.969 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:24.964+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:25.976 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:25.975+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:26.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:26.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:27.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:27.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:28.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:28.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:29.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:29.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:30.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:30.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:31.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:31.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:32.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:32.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:33.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:33.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:34.784 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:34.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:35.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:35.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:36.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:36.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:37.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:37.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:38.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:38.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:39.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:39.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:40.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:40.857+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:41.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:41.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:42.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:42.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:43.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:43.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:44.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:44.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:45.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:45.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:46.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:46.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:47.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:47.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:48.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:48.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:49.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:49.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:50.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:50.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:51.859 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:51.856+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:52.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:52.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:53.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:53.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:54.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:54.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:55.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:55.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:56.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:56.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:57.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:57.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:58.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:58.891+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:14:59.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:14:59.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:00.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:00.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:01.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:01.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:02.835 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:02.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:03.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:03.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:04.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:04.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:05.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:05.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:06.800 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:06.798+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:07.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:07.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:08.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:08.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:09.758 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:09.757+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:10.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:10.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:11.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:11.784+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:12.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:12.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:13.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:13.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:14.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:14.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:15.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:15.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:16.659 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:16.657+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:17.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:17.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:18.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:18.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:19.741 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:19.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:20.397 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:15:20.397 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:15:20.397 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:15:20.398 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:20.398 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:15:20.403 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:15:20.404 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:15:20.404 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:20.404 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:20.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:20.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:21.388 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2208,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:21.416 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2208,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:21.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:21.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:22.755 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:22.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:23.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:23.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:24.416 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:24.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:24.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:24.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:25.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:25.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:26.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:26.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:27.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:27.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:28.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:28.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:29.182 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:29.204 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:29.205 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:15:29.205 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:29.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:15:29.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:29.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:30.205 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2214,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:30.227 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:30.228 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:15:30.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:30.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:31.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:31.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:31.866 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2214,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:31.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:31.889 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:15:32.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:32.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:32.903 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2214,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:15:32.928 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:15:32.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:32.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:15:33.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:33.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:34.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:15:34.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:34.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:35.046 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:15:35.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:35.046 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:35.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:35.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:36.040 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:36.069 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:36.070 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:15:36.070 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:15:36.070 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:36.070 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:15:36.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:36.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:37.069 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2214,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:37.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:37.101 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:15:37.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:37.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:38.116 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2214,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:38.133 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:38.133 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:15:38.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:38.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:39.218 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2214,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:15:39.248 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:15:39.249 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:15:39.249 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:15:39.249 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:15:39.249 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:15:39.250 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:15:39.291 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:15:39.292 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:15:39.292 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:39.292 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:39.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:39.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:40.295 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:40.317 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:40.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:40.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:41.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:41.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:42.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:42.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:43.319 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:43.320 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:43.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:43.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:44.312 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:44.335 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:44.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:44.782+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:45.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:45.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:46.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:46.780+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:47.336 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:47.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:47.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:47.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:48.323 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:48.342 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2214,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:48.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:48.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:49.847 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:49.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:50.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:50.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:51.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:51.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:15:51.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:51.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:52.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:52.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:53.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:53.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:54.245 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:54.268 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:15:54.268 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:15:54.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:54.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:15:54.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:54.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:55.281 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2218,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:55.304 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:15:55.305 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:15:55.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:55.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:56.305 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2218,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:15:56.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:15:56.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:56.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:57.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:57.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:58.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:58.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:15:59.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:15:59.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:00.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:00.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:01.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:01.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:03.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:03.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:03.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:03.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:04.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:04.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:05.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:05.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:07.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:07.041+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:08.082 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:08.081+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:09.079 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:09.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:10.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:10.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:11.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:10.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:12.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:12.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:12.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:12.996+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:13.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:13.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:14.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:14.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:15.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:15.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:16.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:16.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:17.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:17.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:18.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:18.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:19.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:19.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:20.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:20.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:21.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:21.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:22.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:22.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:23.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:23.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:24.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:24.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:25.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:25.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:26.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:26.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:27.982 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:27.980+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:28.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:28.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:29.966 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:29.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:30.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:30.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:31.998 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:31.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:32.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:32.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:33.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:33.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:34.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:34.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:35.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:35.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:36.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:36.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:37.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:37.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:38.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:38.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:39.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:39.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:40.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:40.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:41.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:41.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:42.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:42.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:43.881 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:43.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:44.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:44.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:45.909 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:45.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:46.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:46.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:47.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:47.941+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:48.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:48.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:49.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:49.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:50.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:50.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:51.980 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:51.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:52.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:52.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:53.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:53.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:54.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:54.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:55.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:55.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:56.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:56.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:57.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:57.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:58.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:58.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:16:59.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:16:59.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:00.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:00.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:01.791 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:01.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:02.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:02.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:03.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:03.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:04.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:04.858+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:05.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:05.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:06.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:06.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:07.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:07.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:08.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:08.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:09.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:09.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:10.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:10.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:11.769 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:11.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:12.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:12.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:13.702 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:13.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:14.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:14.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:15.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:15.709+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:16.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:16.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:17.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:17.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:18.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:18.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:19.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:19.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:20.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:20.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:21.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:21.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:22.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:22.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:23.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:23.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:24.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:24.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:25.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:25.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:26.337 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:17:26.337 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:17:26.337 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:17:26.337 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:26.338 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:17:26.343 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:17:26.343 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:17:26.343 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:26.343 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:26.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:26.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:27.312 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:27.334 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:27.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:27.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:28.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:28.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:29.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:29.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:30.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:30.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:30.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:30.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:31.253 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:31.275 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:31.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:31.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:32.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:32.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:33.694 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:33.692+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:34.277 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:34.277 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:34.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:34.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:35.295 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:35.325 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2218,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:35.727 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:35.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:36.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:36.730+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:37.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:37.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:38.327 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:38.327 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:38.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:38.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:39.339 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:39.369 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:39.369 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:17:39.369 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:39.369 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:17:39.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:39.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:40.384 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2220,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:40.405 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:40.405 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:17:40.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:40.590+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:41.396 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2220,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:41.417 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:41.417 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:17:41.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:41.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:42.442 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2220,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:42.472 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:17:42.472 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:42.472 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:17:42.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:42.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:43.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:43.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:43.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:17:44.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:44.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:44.587 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:17:44.587 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:44.587 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:45.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:45.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:45.589 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:45.622 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:45.623 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:17:45.623 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:17:45.623 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:45.623 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:17:46.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:46.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:46.588 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2220,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:46.616 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:46.617 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:17:47.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:47.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:47.628 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2220,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:47.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:47.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:17:48.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:48.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:48.599 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2220,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:17:48.622 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:17:48.622 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:17:48.623 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:17:48.623 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:17:48.623 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:17:48.623 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:17:48.638 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:17:48.638 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:17:48.639 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:48.639 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:49.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:49.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:49.686 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:49.707 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:50.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:50.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:51.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:51.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:52.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:52.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:52.708 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:52.708 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:53.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:53.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:53.707 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:53.737 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:54.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:54.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:55.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:55.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:56.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:56.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:56.738 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:17:56.738 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:17:57.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:57.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:57.731 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:57.754 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2220,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":21,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:17:58.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:58.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:17:59.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:17:59.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:00.491 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:00.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:00.755 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:18:00.755 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:18:01.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:01.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:02.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:02.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:03.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:03.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:03.782 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2224,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:18:03.805 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2224,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:18:03.805 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:18:03.805 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:18:03.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:18:04.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:04.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:04.830 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2224,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:18:04.861 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:18:04.861 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:18:05.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:05.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:05.823 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2224,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:18:05.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:18:06.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:06.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:07.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:07.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:08.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:08.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:09.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:09.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:10.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:10.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:11.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:11.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:12.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:12.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:13.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:13.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:14.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:14.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:15.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:15.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:16.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:16.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:17.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:17.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:18.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:18.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:19.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:19.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:20.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:20.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:21.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:21.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:22.496 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:22.494+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:23.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:23.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:24.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:24.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:25.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:25.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:26.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:26.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:27.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:27.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:28.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:28.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:29.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:29.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:30.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:30.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:31.628 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:31.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:32.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:32.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:33.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:33.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:34.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:34.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:35.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:35.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:36.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:36.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:37.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:37.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:38.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:38.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:39.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:39.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:40.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:40.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:41.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:41.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:42.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:42.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:43.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:43.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:44.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:44.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:45.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:45.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:46.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:46.713+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:47.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:47.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:48.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:48.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:49.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:49.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:50.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:50.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:51.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:51.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:52.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:52.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:53.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:53.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:54.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:54.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:55.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:55.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:56.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:56.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:57.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:57.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:58.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:58.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:18:59.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:18:59.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:00.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:00.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:01.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:01.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:02.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:02.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:03.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:03.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:04.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:04.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:05.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:05.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:06.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:06.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:07.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:07.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:08.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:08.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:09.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:09.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:10.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:10.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:11.729 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:11.728+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:12.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:12.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:13.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:13.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:14.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:14.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:15.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:15.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:16.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:16.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:17.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:17.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:18.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:18.776+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:19.782 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:19.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:20.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:20.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:21.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:21.789+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:22.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:22.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:23.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:23.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:24.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:24.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:25.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:25.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:26.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:26.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:27.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:27.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:28.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:28.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:29.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:29.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:30.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:30.826+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:31.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:31.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:32.850 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:32.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:33.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:33.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:34.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:34.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:35.857 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:19:35.858 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:19:35.858 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:19:35.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:35.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:19:35.865 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:19:35.865 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:19:35.865 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:35.866 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:19:35.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:35.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:36.844 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2224,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:36.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:36.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:36.874 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2224,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:37.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:37.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:38.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:38.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:39.876 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:39.876 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:19:39.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:39.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:40.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:40.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:41.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:41.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:42.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:42.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:43.185 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2228,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:43.208 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2228,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:43.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:43.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:44.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:44.879+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:45.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:45.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:46.208 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:46.208 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:19:46.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:46.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:47.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:47.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:48.808 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:48.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:49.214 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2232,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:49.236 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2232,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:49.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:49.807+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:50.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:50.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:51.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:51.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:52.239 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:52.239 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:19:52.802 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:52.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:53.227 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2234,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:53.251 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2234,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:53.251 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:19:53.252 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:53.252 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:19:53.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:53.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:54.237 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2234,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:19:54.268 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:54.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:19:54.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:54.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:55.289 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2234,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:19:55.320 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:55.320 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:19:55.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:55.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:56.356 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2234,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:19:56.378 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:19:56.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:56.379 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:19:56.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:56.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:57.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:19:57.841 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:57.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:58.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:19:58.516 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:58.516 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:19:58.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:58.863+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:19:59.517 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2234,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:59.549 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2234,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:19:59.549 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:19:59.550 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:19:59.550 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:19:59.550 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:19:59.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:19:59.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:00.584 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2234,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:20:00.608 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:20:00.609 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:20:00.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:00.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:01.603 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2234,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:20:01.633 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:20:01.633 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:20:01.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:01.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:02.626 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2234,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:20:02.657 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:20:02.657 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T11:20:02.657 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:20:02.657 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:20:02.658 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:20:02.658 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:20:02.676 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:20:02.677 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T11:20:02.677 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T11:20:02.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:20:02.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:02.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:03.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:03.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:04.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:04.811+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:05.775 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:05.773+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:06.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:06.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:07.783 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:07.781+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:08.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:08.787+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:09.799 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:09.797+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:10.759 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:10.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:11.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:11.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:12.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:12.809+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:13.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:13.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:14.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:14.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:15.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:15.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:16.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:16.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:17.678 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T11:20:17.679 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T11:20:17.679 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:20:17.679 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:20:17.679 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:20:17.766 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:17.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:18.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:18.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:19.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:19.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:20.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:20.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:21.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:21.695+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:22.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:22.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:23.329 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2238,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:20:23.352 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2238,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:20:23.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:20:23.352 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:20:23.352 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:20:23.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:23.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:24.377 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2238,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:20:24.400 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:20:24.400 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:20:24.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:24.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:25.401 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2238,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:20:25.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:20:25.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:25.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:26.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:26.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:27.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:27.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:28.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:28.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:29.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:29.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:30.635 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:30.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:31.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:31.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:32.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:32.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:33.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:33.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:34.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:34.684+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:35.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:35.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:36.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:36.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:37.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:37.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:38.788 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:38.786+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:39.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:39.760+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:40.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:40.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:41.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:41.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:42.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:42.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:43.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:43.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:44.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:44.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:45.691 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:45.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:46.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:46.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:47.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:47.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:48.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:48.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:49.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:49.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:50.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:50.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:51.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:51.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:52.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:52.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:53.576 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:53.574+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:54.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:54.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:55.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:55.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:56.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:56.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:57.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:57.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:58.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:58.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:20:59.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:20:59.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:00.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:00.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:01.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:01.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:02.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:02.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:03.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:03.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:04.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:04.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:05.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:05.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:06.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:06.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:07.670 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:07.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:08.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:08.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:09.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:09.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:10.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:10.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:11.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:11.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:12.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:12.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:13.671 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:13.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:14.684 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:14.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:15.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:15.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:16.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:16.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:17.761 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:17.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:18.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:18.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:19.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:19.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:20.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:20.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:21.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:21.767+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:22.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:22.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:23.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:23.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:24.866 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:24.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:25.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:25.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:26.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:26.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:27.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:27.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:28.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:28.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:29.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:29.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:30.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:30.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:31.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:31.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:32.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:32.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:34.017 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:34.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:35.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:35.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:36.036 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:36.034+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:37.032 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:37.030+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:38.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:38.008+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:38.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:38.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:39.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:39.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:40.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:40.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:41.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:41.971+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:42.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:42.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:43.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:43.976+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:44.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:44.932+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:45.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:45.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:46.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:46.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:47.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:47.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:49.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:49.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:49.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:49.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:50.957 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:50.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:51.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:51.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:52.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:52.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:53.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:53.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:54.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:54.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:55.432 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:21:55.432 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:21:55.433 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:21:55.433 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:21:55.433 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:21:55.440 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:21:55.440 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T11:21:55.440 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T11:21:55.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:21:55.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:55.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:56.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:56.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:57.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:57.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:58.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:58.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:21:59.970 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:21:59.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:00.939 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:00.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:01.933 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:01.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:02.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:02.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:03.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:03.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:04.887 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:04.885+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:05.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:05.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:06.929 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:06.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:07.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:07.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:08.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:08.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:09.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:09.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:10.442 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T11:22:10.442 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T11:22:10.442 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:22:10.443 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:10.443 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:22:10.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:10.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:11.063 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T11:22:11.061+0000 7f44dffff700 0 --2- 172.21.6.124:0/2526715562 >> [v2:172.21.6.124:3301/0,v1:172.21.6.124:6790/0] conn(0x7f44e013e010 0x7f44e0168100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T11:22:11.691 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2242,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:11.721 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2242,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:11.721 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:22:11.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:11.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:22:11.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:11.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:12.731 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2242,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:12.754 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:12.754 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:22:12.977 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:12.974+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:13.989 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2242,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:13.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:13.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:14.019 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:14.019 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:22:14.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:14.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:15.023 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2242,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:15.045 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:22:15.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:15.046 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:22:15.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:15.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:16.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:22:16.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:16.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:17.179 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:22:17.179 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:17.179 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:22:17.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:17.890+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:18.161 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2242,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:18.193 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2242,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:18.193 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:22:18.193 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:22:18.193 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:18.194 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:22:18.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:18.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:19.204 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2242,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:19.226 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:19.226 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:22:19.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:19.820+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:20.245 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2242,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:20.269 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:20.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:22:20.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:20.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:21.270 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2242,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:21.293 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:22:21.293 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2019-10-01T11:22:21.293 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:22:21.293 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:22:21.294 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:22:21.294 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:22:21.323 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:22:21.323 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T11:22:21.323 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T11:22:21.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:22:21.778 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:21.777+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:22.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:22.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:23.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:23.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:24.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:24.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:25.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:25.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:26.792 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:26.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:27.760 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:27.758+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:28.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:28.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:29.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:29.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:30.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:30.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:31.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:31.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:32.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:32.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:33.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:33.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:34.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:34.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:35.711 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:35.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:36.324 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T11:22:36.325 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T11:22:36.325 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:22:36.326 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:36.326 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:22:36.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:36.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:37.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:37.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:38.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:38.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:39.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:39.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:40.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:40.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:41.725 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:41.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:41.973 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2244,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:41.996 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2244,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:22:41.996 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:22:41.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:41.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:22:42.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:42.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:42.973 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2244,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:22:42.996 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:22:42.996 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:22:43.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:43.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:43.994 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2244,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:22:44.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:22:44.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:44.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:45.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:45.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:46.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:46.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:47.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:47.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:48.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:48.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:49.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:49.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:50.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:50.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:51.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:51.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:52.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:52.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:53.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:53.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:54.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:54.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:55.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:55.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:56.656 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:56.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:57.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:57.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:58.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:58.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:22:59.617 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:22:59.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:00.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:00.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:01.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:01.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:02.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:02.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:03.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:03.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:04.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:04.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:05.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:05.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:06.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:06.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:07.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:07.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:08.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:08.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:09.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:09.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:10.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:10.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:11.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:11.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:12.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:12.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:13.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:13.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:14.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:14.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:15.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:15.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:16.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:16.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:17.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:17.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:18.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:18.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:19.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:19.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:20.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:20.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:21.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:21.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:22.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:22.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:23.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:23.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:24.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:24.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:25.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:25.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:26.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:26.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:27.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:27.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:28.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:28.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:29.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:29.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:30.680 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:30.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:31.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:31.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:32.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:32.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:33.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:33.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:34.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:34.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:35.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:35.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:36.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:36.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:37.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:37.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:38.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:38.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:39.672 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:39.670+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:40.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:40.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:41.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:41.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:42.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:42.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:43.673 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:43.671+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:44.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:44.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:45.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:45.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:46.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:46.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:47.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:47.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:48.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:48.637+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:49.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:49.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:50.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:50.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:51.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:51.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:52.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:52.626+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:53.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:53.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:54.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:54.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:55.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:55.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:56.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:56.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:57.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:57.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:58.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:58.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:23:59.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:23:59.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:00.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:00.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:01.618 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:01.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:02.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:02.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:03.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:03.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:04.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:04.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:05.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:05.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:06.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:06.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:07.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:07.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:08.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:08.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:09.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:09.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:10.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:10.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:11.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:11.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:12.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:12.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:13.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:13.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:14.020 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:24:14.021 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:24:14.021 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:24:14.021 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:24:14.021 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:24:14.029 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:24:14.030 INFO:tasks.mon_thrash:Sending STOP to mon a 2019-10-01T11:24:14.030 INFO:tasks.ceph.mon.a:Sent signal 19 2019-10-01T11:24:14.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:24:14.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:14.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:15.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:15.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:16.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:16.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:17.493 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:17.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:18.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:18.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:19.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:19.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:20.460 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:20.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:21.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:21.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:22.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:22.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:23.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:23.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:24.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:24.454+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:25.412 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:25.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:26.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:26.382+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:27.429 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:27.427+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:28.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:28.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:29.031 INFO:tasks.mon_thrash:Sending CONT to mon a 2019-10-01T11:24:29.031 INFO:tasks.ceph.mon.a:Sent signal 18 2019-10-01T11:24:29.031 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:24:29.032 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:29.032 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:24:29.464 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:29.462+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:30.048 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2250,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:24:30.076 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2250,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:24:30.076 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:24:30.077 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:30.077 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:24:30.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:30.414+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:31.068 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2250,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:24:31.099 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:31.099 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:24:31.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:31.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:32.092 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2250,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:24:32.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:32.114 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:24:32.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:32.344+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:33.111 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2250,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:24:33.141 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:24:33.141 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:33.141 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:24:33.342 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:33.340+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:34.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:24:34.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:34.380+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:35.250 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:24:35.251 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:35.251 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:24:35.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:35.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:36.206 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2250,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:24:36.238 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2250,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:24:36.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:24:36.238 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:24:36.238 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:36.238 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:24:36.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:36.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:37.219 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2250,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:24:37.241 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:37.241 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:24:37.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:37.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:38.234 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2250,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:24:38.257 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:38.257 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:24:38.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:38.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:39.264 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2250,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:24:39.280 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:39.278+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:39.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:24:39.294 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2019-10-01T11:24:39.294 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:24:39.294 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:24:39.295 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:24:39.295 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:24:39.321 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:24:39.322 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T11:24:39.322 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T11:24:39.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:24:40.306 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:40.304+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:41.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:41.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:42.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:42.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:43.271 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:43.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:44.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:44.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:45.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:45.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:46.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:46.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:47.304 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:47.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:48.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:48.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:49.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:49.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:50.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:50.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:51.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:51.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:52.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:52.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:53.353 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:53.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:54.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:54.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:54.323 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T11:24:54.323 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T11:24:54.323 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:24:54.324 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:24:54.324 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:24:55.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:55.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:56.305 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:56.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:57.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:57.333+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:58.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:58.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:24:59.386 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:24:59.384+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:00.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:00.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:01.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:01.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:01.534 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2252,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:25:01.566 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2252,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:25:01.566 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:25:01.566 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:25:01.567 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:25:02.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:02.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:02.574 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2252,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:25:02.604 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:25:02.604 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:25:03.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:03.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:03.599 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2252,"quorum":[0,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:25:03.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:25:04.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:04.363+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:05.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:05.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:06.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:06.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:07.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:07.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:08.396 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:08.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:09.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:09.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:10.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:10.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:11.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:11.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:12.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:12.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:13.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:13.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:14.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:14.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:15.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:15.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:16.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:16.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:17.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:17.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:18.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:18.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:19.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:19.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:20.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:20.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:21.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:21.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:22.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:22.489+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:23.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:23.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:24.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:24.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:25.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:25.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:26.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:26.493+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:27.495 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:27.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:28.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:28.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:29.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:29.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:30.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:30.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:31.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:31.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:32.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:32.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:33.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:33.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:34.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:34.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:35.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:35.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:36.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:36.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:37.442 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:37.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:38.457 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:38.455+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:39.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:39.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:40.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:40.475+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:41.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:41.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:42.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:42.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:43.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:43.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:44.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:44.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:45.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:45.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:46.531 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:46.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:47.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:47.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:48.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:48.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:49.542 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:49.540+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:50.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:50.502+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:51.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:51.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:52.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:52.517+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:53.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:53.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:54.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:54.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:55.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:55.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:56.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:56.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:57.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:57.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:58.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:58.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:25:59.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:25:59.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:00.589 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:00.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:01.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:01.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:02.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:02.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:03.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:03.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:04.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:04.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:05.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:05.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:06.625 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:06.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:07.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:07.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:08.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:08.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:09.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:09.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:10.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:10.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:11.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:11.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:12.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:12.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:13.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:13.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:14.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:14.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:15.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:15.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:16.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:16.619+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:17.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:17.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:18.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:18.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:19.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:19.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:20.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:20.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:21.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:21.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:22.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:22.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:23.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:23.553+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:24.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:24.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:25.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:25.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:26.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:26.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:27.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:27.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:28.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:28.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:29.440 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:29.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:30.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:30.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:31.443 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:31.441+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:32.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:32.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:33.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:33.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:33.622 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:26:33.622 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:26:33.622 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:26:33.622 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:33.622 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:26:33.630 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:26:33.630 INFO:tasks.mon_thrash:Sending STOP to mon c 2019-10-01T11:26:33.630 INFO:tasks.ceph.mon.c:Sent signal 19 2019-10-01T11:26:33.631 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:26:34.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:34.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:35.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:35.387+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:36.403 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:36.401+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:37.362 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:37.360+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:38.401 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:38.399+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:39.390 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:39.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:40.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:40.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:41.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:41.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:42.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:42.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:43.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:43.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:44.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:44.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:45.476 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:45.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:46.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:46.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:47.447 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:47.446+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:48.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:48.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:48.631 INFO:tasks.mon_thrash:Sending CONT to mon c 2019-10-01T11:26:48.632 INFO:tasks.ceph.mon.c:Sent signal 18 2019-10-01T11:26:48.632 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:26:48.632 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:48.632 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:26:49.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:49.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:49.628 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:26:49.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:26:49.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:26:49.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:49.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:26:50.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:50.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:50.652 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2256,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:26:50.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:50.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:26:51.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:51.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:51.679 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2256,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:26:51.712 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:51.712 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:26:52.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:52.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:52.714 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2256,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:26:52.731 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:26:52.731 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:52.732 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:26:53.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:53.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:53.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:26:54.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:54.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:54.893 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:26:54.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:54.894 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:26:55.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:55.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:55.873 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:26:55.895 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:26:55.895 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:26:55.895 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:26:55.895 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:55.896 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:26:56.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:56.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:56.871 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2256,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:26:56.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:56.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:26:57.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:57.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:57.847 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2256,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:26:57.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:57.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:26:58.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:58.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:58.867 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2256,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:26:58.896 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:26:58.896 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:26:58.897 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:26:58.897 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:26:58.897 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:26:58.897 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:26:58.910 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:26:58.910 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:26:58.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:26:58.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:26:59.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:26:59.586+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:26:59.976 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:00.004 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:00.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:00.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:01.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:01.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:02.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:02.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:03.004 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:27:03.005 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:27:03.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:03.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:03.992 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:04.015 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2256,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:04.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:04.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:05.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:05.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:06.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:06.472+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:07.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:27:07.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:27:07.482 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:07.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:08.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:08.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:09.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:09.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:10.414 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:10.410+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:11.389 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:11.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:12.406 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:12.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:13.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:13.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:13.571 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2260,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:13.601 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2260,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:27:13.601 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:27:13.602 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:27:13.602 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:27:14.434 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:14.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:14.595 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2260,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:27:14.625 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:27:14.626 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:27:15.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:15.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:15.621 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2260,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:27:15.650 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:27:16.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:16.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:17.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:17.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:18.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:18.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:19.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:19.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:20.427 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:20.425+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:21.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:21.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:22.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:22.439+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:23.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:23.457+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:24.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:24.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:25.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:25.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:26.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:26.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:27.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:27.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:28.422 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:28.420+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:29.446 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:29.444+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:30.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:30.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:31.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:31.483+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:32.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:32.491+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:33.465 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:33.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:34.503 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:34.501+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:35.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:35.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:36.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:36.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:37.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:37.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:38.516 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:38.514+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:39.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:39.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:40.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:40.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:41.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:41.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:42.456 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:42.451+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:43.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:43.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:44.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:44.486+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:45.480 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:45.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:46.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:46.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:47.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:47.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:48.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:48.479+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:49.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:49.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:50.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:50.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:51.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:51.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:52.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:52.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:53.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:53.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:54.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:54.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:55.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:55.470+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:56.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:56.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:57.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:57.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:58.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:58.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:27:59.481 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:27:59.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:00.513 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:00.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:01.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:01.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:02.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:02.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:03.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:03.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:04.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:04.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:05.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:05.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:06.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:06.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:07.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:07.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:08.517 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:08.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:09.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:09.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:10.452 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:10.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:11.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:11.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:12.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:12.434+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:13.441 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:13.440+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:14.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:14.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:15.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:15.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:16.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:16.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:17.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:17.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:18.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:18.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:19.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:19.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:20.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:20.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:21.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:21.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:22.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:22.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:23.614 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:23.612+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:24.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:24.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:25.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:25.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:26.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:26.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:27.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:27.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:28.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:28.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:29.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:29.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:30.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:30.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:31.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:31.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:32.588 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:32.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:33.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:33.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:34.645 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:34.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:35.667 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:35.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:36.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:36.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:37.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:37.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:38.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:38.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:39.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:39.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:40.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:40.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:41.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:41.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:42.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:42.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:43.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:43.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:44.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:44.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:45.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:45.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:45.651 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:28:45.652 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:28:45.652 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:28:45.652 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:45.652 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:28:45.659 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:28:45.660 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:28:45.660 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:45.660 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:28:46.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:46.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:46.643 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2260,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:28:46.666 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2260,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:28:47.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:47.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:48.466 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:48.463+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:49.435 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:49.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:49.667 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:49.668 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:28:50.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:50.459+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:51.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:51.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:52.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:52.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:53.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:53.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:54.485 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:54.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:54.617 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:28:54.639 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:28:54.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:28:54.640 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:54.640 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:28:55.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:55.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:55.611 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2266,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:28:55.626 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:55.626 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:28:56.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:56.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:56.633 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2266,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:28:56.655 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:56.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:28:57.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:57.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:57.666 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2266,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:28:57.696 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:28:57.697 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:57.697 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:28:58.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:58.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:58.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:28:59.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:28:59.615+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:28:59.788 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:28:59.789 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:28:59.789 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:29:00.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:00.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:00.763 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:00.793 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:00.793 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:29:00.794 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:29:00.794 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:00.794 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:29:01.577 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:01.575+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:01.799 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2266,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:29:01.829 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:01.829 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:29:02.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:02.545+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:02.859 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2266,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:29:02.889 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:02.890 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:29:03.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:03.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:03.905 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2266,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:29:03.927 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:29:03.927 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:29:03.927 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:29:03.928 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:29:03.928 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:29:03.928 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:29:03.946 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:29:03.947 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:29:03.947 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:03.947 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:29:04.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:04.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:04.946 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:04.969 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:05.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:05.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:06.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:06.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:07.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:07.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:07.970 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:07.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:29:08.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:08.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:08.946 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:08.968 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:09.507 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:09.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:10.501 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:10.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:11.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:11.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:11.969 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:11.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:29:12.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:12.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:12.971 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:12.993 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2266,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:13.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:13.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:14.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:14.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:15.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:15.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:15.994 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:15.995 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:29:16.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:16.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:17.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:17.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:18.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:18.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:18.737 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:18.759 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:29:18.760 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:29:18.760 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:18.760 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:29:19.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:19.654+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:19.679 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2268,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:29:19.709 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:29:19.709 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:29:20.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:20.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:20.632 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2268,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:29:20.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:29:21.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:21.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:22.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:22.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:23.674 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:23.672+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:24.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:24.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:25.621 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:25.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:26.641 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:26.639+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:27.678 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:27.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:28.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:28.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:29.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:29.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:30.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:30.656+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:31.692 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:31.690+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:32.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:32.688+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:33.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:33.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:34.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:34.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:35.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:35.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:36.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:36.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:37.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:37.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:38.604 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:38.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:39.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:39.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:40.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:40.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:41.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:41.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:42.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:42.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:43.587 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:43.585+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:44.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:44.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:45.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:45.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:46.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:46.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:47.554 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:47.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:48.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:48.506+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:49.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:49.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:50.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:50.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:51.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:51.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:52.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:52.533+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:53.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:53.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:54.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:54.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:55.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:55.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:56.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:56.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:57.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:57.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:58.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:58.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:29:59.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:29:59.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:00.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:00.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:01.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:01.616+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:02.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:02.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:03.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:03.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:04.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:04.622+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:05.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:05.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:06.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:06.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:07.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:07.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:08.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:08.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:09.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:09.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:10.630 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:10.628+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:11.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:11.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:12.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:12.698+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:13.676 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:13.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:14.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:14.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:15.662 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:15.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:16.636 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:16.634+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:17.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:17.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:18.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:18.595+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:19.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:19.644+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:20.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:20.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:21.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:21.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:22.535 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:22.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:23.533 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:23.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:24.529 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:24.527+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:25.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:25.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:26.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:26.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:27.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:27.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:28.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:28.474+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:29.439 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:29.438+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:30.474 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:30.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:31.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:31.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:32.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:32.558+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:33.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:33.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:34.632 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:34.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:35.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:35.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:36.611 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:36.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:37.601 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:37.599+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:38.610 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:38.608+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:39.640 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:39.638+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:40.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:40.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:41.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:41.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:42.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:42.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:43.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:43.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:44.657 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:44.655+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:45.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:45.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:46.580 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:46.578+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:47.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:47.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:48.569 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:48.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:49.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:49.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:50.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:50.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:50.664 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:30:50.664 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:30:50.664 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:30:50.665 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:30:50.665 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:30:50.671 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:30:50.672 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:30:50.672 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:30:50.672 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:30:51.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:51.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:51.690 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:30:51.720 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:30:52.591 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:52.589+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:53.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:53.566+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:54.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:54.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:54.722 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:30:54.722 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:30:55.551 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:55.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:55.701 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:30:55.722 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:30:56.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:56.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:57.508 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:57.507+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:58.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:58.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:58.723 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:30:58.723 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:30:59.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:30:59.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:30:59.659 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:30:59.691 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:00.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:00.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:01.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:01.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:02.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:02.478+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:02.692 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:02.692 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:03.484 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:03.482+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:03.712 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:03.734 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2268,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:04.490 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:04.488+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:05.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:05.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:06.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:06.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:06.736 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:06.736 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:07.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:07.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:08.623 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:08.621+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:09.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:09.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:10.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:10.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:11.563 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:11.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:12.595 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:12.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:13.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:13.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:14.585 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:14.583+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:15.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:15.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:16.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:16.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:17.049 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:17.070 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:17.070 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:31:17.071 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:17.071 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:31:17.522 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:17.520+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:18.133 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2282,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:31:18.161 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:18.162 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:31:18.566 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:18.564+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:19.178 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2282,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:19.200 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:19.200 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:31:19.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:19.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:20.207 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2282,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:20.237 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:31:20.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:20.237 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:31:20.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:20.569+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:21.342 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:31:21.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:21.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:22.343 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:31:22.344 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:22.344 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:22.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:22.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:23.350 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:23.383 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:23.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:31:23.383 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:31:23.383 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:23.383 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:31:23.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:23.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:24.369 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2282,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":3}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:31:24.392 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:24.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:31:24.459 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:24.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:25.388 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2282,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:25.412 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:25.413 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:31:25.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:25.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:26.422 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2282,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:26.444 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:31:26.445 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:31:26.445 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:31:26.445 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:31:26.445 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:31:26.445 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:31:26.464 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:31:26.464 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:31:26.465 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:26.465 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:26.498 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:26.496+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:27.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:27.508+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:27.517 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:27.540 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:28.462 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:28.460+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:29.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:29.431+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:30.444 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:30.442+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:30.541 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:30.541 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:31.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:31.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:31.533 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:31.556 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2282,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:32.450 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:32.448+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:33.468 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:33.466+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:34.445 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:34.443+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:34.557 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:34.557 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:31:35.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:35.481+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:36.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:36.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:37.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:37.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:38.478 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:38.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:39.463 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:39.461+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:40.449 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:40.447+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:41.473 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:41.471+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:41.665 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:41.695 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:31:41.695 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:31:41.695 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:41.696 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:31:42.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:42.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:42.707 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2286,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:42.739 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:31:42.739 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:31:43.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:43.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:43.716 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2286,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:31:43.747 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:31:44.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:44.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:45.489 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:45.487+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:46.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:46.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:47.467 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:47.465+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:48.477 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:48.476+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:49.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:49.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:50.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:50.505+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:51.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:51.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:52.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:52.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:53.547 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:53.546+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:54.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:54.563+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:55.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:55.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:56.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:56.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:57.647 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:57.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:58.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:58.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:31:59.704 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:31:59.702+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:00.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:00.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:01.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:01.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:02.728 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:02.726+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:03.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:03.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:04.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:04.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:05.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:05.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:06.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:06.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:07.688 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:07.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:08.698 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:08.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:09.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:09.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:10.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:10.678+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:11.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:11.667+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:12.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:12.663+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:13.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:13.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:14.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:14.647+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:15.624 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:15.623+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:16.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:16.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:17.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:17.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:18.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:18.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:19.690 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:19.689+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:20.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:20.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:21.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:21.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:22.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:22.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:23.746 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:23.744+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:24.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:24.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:25.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:25.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:26.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:26.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:27.737 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:27.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:28.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:28.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:29.697 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:29.696+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:30.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:30.721+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:31.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:31.763+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:32.771 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:32.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:33.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:33.815+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:34.790 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:34.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:35.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:35.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:36.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:36.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:37.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:37.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:38.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:38.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:39.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:39.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:40.958 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:40.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:41.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:41.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:42.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:42.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:43.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:43.949+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:44.979 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:44.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:45.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:45.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:46.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:46.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:47.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:47.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:48.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:48.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:49.892 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:49.888+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:50.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:50.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:51.941 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:51.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:52.908 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:52.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:53.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:53.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:54.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:54.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:55.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:55.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:57.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:56.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:57.975 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:57.973+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:58.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:58.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:32:59.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:32:59.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:00.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:00.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:01.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:01.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:02.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:02.851+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:03.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:03.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:04.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:04.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:05.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:05.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:06.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:06.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:07.700 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:07.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:08.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:08.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:09.606 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:09.604+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:10.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:10.611+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:11.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:11.607+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:12.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:12.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:13.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:13.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:13.748 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:33:13.749 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:33:13.749 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:33:13.749 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:13.749 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:33:13.755 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:33:13.755 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:33:13.755 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:13.755 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:14.568 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:14.567+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:14.727 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:14.749 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:15.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:15.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:16.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:16.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:17.552 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:17.550+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:17.749 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:17.750 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:18.575 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:18.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:18.764 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:18.782 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:19.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:19.551+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:20.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:20.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:21.520 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:21.518+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:21.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:21.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:22.479 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:22.477+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:22.739 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:22.764 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2286,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:23.455 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:23.453+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:24.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:24.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:25.523 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:25.521+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:25.765 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:25.765 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:26.486 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:26.484+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:26.781 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:26.810 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":3,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:26.810 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:33:26.810 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:26.811 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:33:27.512 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:27.511+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:27.761 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2288,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:27.784 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:27.784 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:33:28.506 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:28.504+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:28.785 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2288,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:28.806 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:28.806 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:33:29.527 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:29.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:29.756 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2288,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:29.777 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:33:29.778 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:29.778 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:33:30.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:30.561+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:30.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:33:31.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:31.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:31.870 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:33:31.870 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:31.870 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:32.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:32.573+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:32.836 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:32.858 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:32.858 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:33:32.858 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:33:32.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:32.859 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:33:33.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:33.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:33.858 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2288,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:33.890 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:33.891 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:33:34.557 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:34.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:35.115 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2288,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:35.136 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:35.136 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:33:35.514 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:35.512+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:36.149 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2288,"quorum":[0,1,2],"quorum_age":13,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:36.172 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:33:36.172 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:33:36.172 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:33:36.172 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:33:36.173 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:33:36.173 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:33:36.185 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:33:36.185 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:33:36.185 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:36.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:36.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:36.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:37.228 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:37.263 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:37.511 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:37.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:38.509 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:38.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:39.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:39.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:40.263 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:40.264 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:40.499 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:40.497+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:41.268 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:41.298 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":18,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:41.546 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:41.543+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:42.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:42.525+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:43.518 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:43.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:44.300 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:44.300 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:44.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:44.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:45.306 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:45.336 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2288,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":22,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:45.572 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:45.570+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:46.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:46.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:47.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:47.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:48.336 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:48.336 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:33:48.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:48.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:49.599 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:49.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:50.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:50.625+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:51.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:51.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:52.395 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2292,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:52.417 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2292,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:33:52.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:33:52.418 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:52.418 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:33:52.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:52.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:53.424 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2292,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:53.456 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:33:53.456 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:33:53.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:53.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:54.414 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2292,"quorum":[0,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:33:54.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:33:54.652 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:54.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:55.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:55.630+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:56.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:56.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:57.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:57.633+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:58.608 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:58.606+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:33:59.631 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:33:59.629+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:00.651 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:00.649+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:01.699 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:01.697+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:02.730 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:02.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:03.731 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:03.729+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:04.716 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:04.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:05.675 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:05.673+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:06.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:06.700+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:07.717 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:07.715+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:08.743 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:08.741+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:09.709 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:09.706+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:10.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:10.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:11.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:11.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:12.714 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:12.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:13.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:13.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:14.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:14.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:15.736 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:15.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:16.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:16.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:17.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:17.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:18.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:18.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:19.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:19.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:20.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:20.750+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:21.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:21.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:22.780 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:22.778+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:23.776 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:23.774+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:24.826 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:24.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:25.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:25.824+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:26.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:26.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:27.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:27.865+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:28.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:28.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:29.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:29.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:30.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:30.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:31.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:31.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:32.851 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:32.848+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:33.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:33.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:34.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:34.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:35.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:35.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:36.845 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:36.843+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:37.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:37.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:38.889 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:38.887+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:39.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:39.923+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:40.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:40.959+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:41.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:41.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:42.956 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:42.955+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:43.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:43.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:44.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:44.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:45.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:45.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:46.921 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:46.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:47.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:47.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:48.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:48.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:49.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:49.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:50.867 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:50.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:51.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:51.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:52.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:52.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:53.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:53.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:54.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:54.860+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:55.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:55.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:56.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:56.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:57.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:57.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:58.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:58.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:34:59.853 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:34:59.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:00.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:00.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:01.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:01.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:02.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:02.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:03.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:03.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:04.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:04.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:05.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:05.900+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:06.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:06.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:07.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:07.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:08.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:08.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:09.905 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:09.903+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:10.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:10.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:11.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:11.894+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:12.931 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:12.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:13.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:13.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:14.916 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:14.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:15.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:15.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:16.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:16.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:17.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:17.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:18.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:18.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:19.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:19.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:20.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:20.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:21.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:21.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:22.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:22.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:23.810 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:23.808+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:24.437 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:35:24.437 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:35:24.437 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:35:24.437 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:24.438 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:35:24.443 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:35:24.444 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:35:24.444 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:24.444 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:35:24.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:24.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:25.795 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:25.794+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:26.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:26.803+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:27.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:27.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:28.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:28.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:29.899 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:29.897+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:30.161 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2296,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:30.193 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2296,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:30.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:30.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:31.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:31.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:33.029 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:33.027+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:33.194 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:33.194 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:35:33.987 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:33.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:34.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:34.951+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:35.789 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:35.812 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:35.812 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:35:35.813 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:35.813 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:35:35.927 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:35.925+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:36.936 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:36.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:37.072 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2304,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:37.094 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:37.094 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:35:37.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:37.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:38.096 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2304,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:38.118 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:38.119 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:35:38.991 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:38.989+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:39.078 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2304,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:39.101 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:35:39.101 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:39.101 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:35:40.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:39.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:40.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:35:41.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:41.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:41.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:35:41.201 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:41.201 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:35:42.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:42.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:42.165 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:42.187 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2304,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:35:42.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:35:42.188 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:35:42.188 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:42.188 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:35:43.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:43.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:43.186 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2304,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:43.217 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:43.218 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:35:44.118 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:44.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:44.242 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2304,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:44.271 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:35:44.271 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:35:45.124 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:45.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:45.261 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2304,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:35:45.291 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:35:45.291 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2019-10-01T11:35:45.291 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:35:45.291 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:35:45.292 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:35:45.292 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:35:45.310 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:35:45.311 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T11:35:45.311 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T11:35:45.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:35:46.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:46.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:47.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:47.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:48.063 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:48.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:49.105 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:49.103+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:50.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:50.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:51.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:51.195+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:52.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:52.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:53.220 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:53.218+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:54.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:54.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:55.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:55.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:56.235 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:56.233+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:57.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:57.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:58.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:58.268+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:35:59.262 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:35:59.260+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:00.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:00.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:00.475 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T11:36:00.476 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T11:36:00.476 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:36:00.476 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:36:00.476 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:36:01.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:01.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:02.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:02.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:03.178 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:03.176+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:04.184 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:04.172+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:05.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:05.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:06.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:06.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:06.142 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2306,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:36:06.166 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2306,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:36:06.167 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:36:06.167 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:36:06.167 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:36:07.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:07.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:07.083 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2306,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:36:07.114 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:36:07.115 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:36:08.080 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2306,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:36:08.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:08.093+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:08.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:36:09.117 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:09.115+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:10.109 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:10.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:11.071 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:11.068+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:12.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:12.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:13.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:13.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:14.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:13.999+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:15.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:15.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:16.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:16.023+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:17.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:17.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:18.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:18.091+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:19.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:19.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:20.115 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:20.113+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:21.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:21.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:22.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:22.209+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:23.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:23.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:24.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:24.249+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:25.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:25.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:26.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:26.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:27.246 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:27.244+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:28.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:28.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:29.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:29.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:30.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:30.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:31.199 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:31.197+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:32.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:32.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:33.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:33.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:34.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:34.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:35.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:35.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:36.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:36.062+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:37.093 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:37.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:38.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:38.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:39.147 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:39.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:40.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:40.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:41.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:41.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:42.248 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:42.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:43.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:43.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:44.263 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:44.261+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:45.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:45.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:46.296 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:46.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:47.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:47.276+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:48.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:48.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:49.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:49.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:50.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:50.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:51.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:51.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:52.328 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:52.327+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:53.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:53.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:54.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:54.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:55.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:55.350+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:56.357 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:56.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:57.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:57.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:58.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:58.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:36:59.398 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:36:59.396+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:00.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:00.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:01.365 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:01.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:02.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:02.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:03.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:03.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:04.384 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:04.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:05.360 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:05.358+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:06.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:06.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:07.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:07.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:08.322 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:08.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:09.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:09.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:10.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:10.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:11.295 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:11.293+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:12.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:12.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:13.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:13.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:14.219 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:14.217+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:15.175 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:15.173+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:16.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:16.215+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:17.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:17.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:18.260 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:18.258+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:19.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:19.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:20.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:20.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:21.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:21.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:22.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:22.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:23.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:23.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:24.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:24.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:25.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:25.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:26.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:26.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:27.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:27.319+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:28.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:28.289+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:29.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:29.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:30.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:30.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:31.314 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:31.303+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:32.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:32.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:33.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:33.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:34.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:34.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:35.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:35.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:36.341 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:36.339+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:37.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:37.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:38.103 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:37:38.104 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:37:38.104 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:37:38.104 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:37:38.104 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:37:38.110 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:37:38.110 INFO:tasks.mon_thrash:Sending STOP to mon b 2019-10-01T11:37:38.110 INFO:tasks.ceph.mon.b:Sent signal 19 2019-10-01T11:37:38.111 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2019-10-01T11:37:38.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:38.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:39.320 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:39.318+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:40.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:40.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:41.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:41.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:42.338 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:42.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:43.303 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:43.302+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:44.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:44.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:45.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:45.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:46.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:46.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:47.292 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:47.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:48.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:48.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:49.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:49.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:50.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:50.294+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:51.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:51.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:52.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:52.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:53.111 INFO:tasks.mon_thrash:Sending CONT to mon b 2019-10-01T11:37:53.112 INFO:tasks.ceph.mon.b:Sent signal 18 2019-10-01T11:37:53.112 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:37:53.112 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:37:53.112 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:37:53.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:53.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:54.313 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:54.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:55.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:55.328+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:56.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:56.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:57.352 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:57.351+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:58.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:58.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:37:59.286 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:37:59.284+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:00.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:00.320+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:01.329 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:01.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:02.319 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:02.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:03.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:03.347+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:04.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:04.337+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:05.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:05.323+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:06.289 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:06.287+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:07.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:07.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:08.298 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:08.296+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:09.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:09.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:09.800 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:09.830 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:09.830 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:38:09.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:09.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:38:10.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:10.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:10.823 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2328,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:10.855 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:10.856 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:38:11.299 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:11.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:11.873 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2328,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:11.905 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:11.905 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:38:12.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:12.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:12.898 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2328,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:12.928 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:38:12.928 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:12.928 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:38:13.240 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:13.238+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:14.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:38:14.206 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:14.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:15.038 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:38:15.039 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:15.039 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:38:15.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:15.220+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:16.025 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:16.054 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:16.054 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:38:16.055 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:38:16.055 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:16.055 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:38:16.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:16.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:17.019 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2328,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:17.051 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:17.051 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:38:17.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:17.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:18.030 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2328,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:18.059 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:18.060 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:38:18.277 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:18.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:19.021 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2328,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:19.051 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:38:19.051 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:38:19.051 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:38:19.051 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:38:19.052 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:38:19.052 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:38:19.067 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:38:19.067 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:38:19.068 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:19.068 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:38:19.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:19.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:20.146 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:20.168 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":10,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:20.222 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:20.221+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:21.267 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:21.265+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:22.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:22.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:23.169 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:23.169 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:38:23.245 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:23.243+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:24.148 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:24.170 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":14,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:24.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:24.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:25.241 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:25.239+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:26.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:26.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:27.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:27.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:27.171 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:27.171 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:38:28.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:28.138+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:28.161 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:28.183 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2328,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:29.126 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:29.124+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:30.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:30.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:31.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:31.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:31.184 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:31.185 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:38:32.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:32.162+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:33.181 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:33.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:34.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:34.145+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:34.156 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2332,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:34.186 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2332,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:38:34.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:38:34.186 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:34.186 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:38:35.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:35.104+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:35.182 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2332,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:35.205 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:38:35.205 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:38:36.095 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:36.092+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:36.108 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2332,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:38:36.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:38:37.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:37.064+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:38.022 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:38.019+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:38.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:38.978+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:39.959 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:39.956+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:40.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:40.934+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:41.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:41.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:42.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:42.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:43.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:43.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:44.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:44.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:45.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:45.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:47.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:47.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:48.037 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:48.035+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:49.012 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:49.010+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:49.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:49.991+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:50.978 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:50.977+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:52.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:52.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:52.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:52.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:54.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:54.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:55.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:55.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:55.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:55.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:56.953 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:56.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:57.906 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:57.904+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:58.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:58.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:38:59.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:38:59.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:00.875 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:00.873+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:01.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:01.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:02.863 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:02.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:03.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:03.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:04.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:04.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:05.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:05.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:06.862 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:06.861+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:07.861 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:07.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:08.896 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:08.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:09.856 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:09.854+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:10.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:10.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:11.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:11.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:12.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:12.870+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:13.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:13.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:14.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:14.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:15.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:15.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:16.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:16.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:17.888 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:17.886+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:18.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:18.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:19.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:19.869+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:20.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:20.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:21.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:21.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:22.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:22.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:23.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:23.867+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:24.874 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:24.872+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:25.897 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:25.895+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:26.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:26.921+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:27.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:27.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:28.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:28.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:29.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:29.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:30.968 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:30.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:32.015 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:32.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:33.010 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:33.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:34.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:34.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:35.030 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:35.028+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:35.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:35.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:37.006 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:37.004+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:37.965 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:37.962+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:38.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:38.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:39.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:39.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:40.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:40.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:41.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:41.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:42.900 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:42.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:43.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:43.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:44.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:44.940+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:45.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:45.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:46.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:46.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:47.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:47.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:48.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:48.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:49.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:49.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:50.963 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:50.961+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:51.919 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:51.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:52.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:52.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:53.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:53.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:54.870 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:54.868+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:55.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:55.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:56.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:56.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:57.849 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:57.847+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:58.860 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:58.859+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:39:59.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:39:59.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:00.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:00.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:01.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:01.832+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:02.804 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:02.802+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:03.854 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:03.852+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:04.904 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:04.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:05.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:05.917+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:06.138 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:40:06.138 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:40:06.139 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:40:06.139 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:06.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:40:06.146 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:40:06.146 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:40:06.147 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:06.147 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:06.922 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:06.920+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:07.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:07.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:08.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:08.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:09.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:09.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:11.013 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:11.011+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:11.912 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2336,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:11.951 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2336,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:12.000 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:11.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:13.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:13.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:14.075 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:14.072+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:14.952 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:14.952 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:15.033 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:15.031+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:16.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:16.002+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:17.021 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:17.013+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:17.139 INFO:teuthology.orchestra.run.mira063.stderr:2019-10-01T11:40:17.137+0000 7fb93b59e700 0 --2- 172.21.6.124:0/522862926 >> v2:172.21.6.124:3300/0 conn(0x7fb93c1426d0 0x7fb93c144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-10-01T11:40:17.756 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:17.788 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:17.788 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:40:17.788 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:17.788 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:40:18.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:18.006+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:18.801 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2344,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:18.832 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:19.089 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:40:19.090 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:19.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:19.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:19.997+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:20.106 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2344,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:20.129 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:20.129 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:40:21.039 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:21.037+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:21.131 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2344,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:21.154 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:40:21.154 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:21.155 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:40:22.078 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:22.076+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:22.302 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:40:23.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:23.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:23.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:40:23.303 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:23.303 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:24.108 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:24.100+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:24.304 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:24.335 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:24.335 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:40:24.335 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:40:24.335 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:24.335 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:40:25.064 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:25.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:25.364 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2344,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:25.396 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:25.396 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:40:26.020 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:26.018+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:26.414 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2344,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:26.446 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:26.446 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:40:26.973 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:26.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:27.466 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2344,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:27.496 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:40:27.496 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:40:27.496 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:40:27.497 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:40:27.497 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:40:27.497 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:40:27.529 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:40:27.529 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:40:27.529 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:27.530 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:27.961 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:27.960+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:28.604 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:28.626 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:29.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:29.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:29.986 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:29.984+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:30.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:30.965+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:31.627 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:31.627 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:32.005 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:32.003+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:32.627 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:32.650 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:33.041 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:33.039+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:34.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:34.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:35.069 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:35.067+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:35.651 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:35.651 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:36.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:36.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:36.646 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:36.669 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2344,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:37.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:37.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:38.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:38.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:39.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:39.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:39.670 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:39.670 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:40:40.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:40.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:41.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:41.084+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:42.085 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:42.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:42.920 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2348,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:42.950 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2348,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:40:42.950 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:40:42.950 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:42.950 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:40:43.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:43.082+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:43.971 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2348,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:40:44.004 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:40:44.004 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:40:44.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:44.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:45.010 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2348,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:40:45.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:40:45.133 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:45.131+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:46.086 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:46.083+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:47.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:47.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:48.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:48.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:49.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:49.015+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:50.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:50.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:51.024 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:51.022+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:51.983 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:51.981+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:52.943 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:52.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:53.964 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:53.963+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:54.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:54.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:55.937 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:55.935+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:56.940 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:56.938+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:57.913 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:57.911+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:58.886 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:58.884+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:40:59.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:40:59.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:00.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:00.901+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:01.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:01.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:02.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:02.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:03.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:03.833+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:04.819 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:04.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:05.809 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:05.806+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:06.834 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:06.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:07.837 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:07.835+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:08.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:08.837+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:09.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:09.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:10.815 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:10.813+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:11.839 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:11.838+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:12.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:12.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:13.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:13.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:14.828 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:14.827+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:15.844 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:15.842+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:16.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:16.846+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:17.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:17.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:18.829 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:18.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:19.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:19.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:20.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:20.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:21.806 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:21.804+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:22.823 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:22.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:23.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:23.788+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:24.768 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:24.766+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:25.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:25.753+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:26.767 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:26.765+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:27.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:27.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:28.832 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:28.830+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:29.846 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:29.844+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:30.822 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:30.821+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:31.812 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:31.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:32.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:32.850+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:33.818 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:33.816+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:34.816 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:34.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:35.831 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:35.829+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:36.824 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:36.822+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:37.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:37.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:38.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:38.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:39.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:39.875+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:40.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:40.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:41.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:41.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:42.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:42.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:43.872 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:43.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:44.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:44.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:45.930 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:45.929+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:46.898 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:46.896+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:47.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:47.928+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:48.934 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:48.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:49.984 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:49.982+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:50.996 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:50.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:52.001 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:51.998+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:52.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:52.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:53.971 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:53.967+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:54.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:54.970+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:55.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:55.969+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:56.938 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:56.936+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:57.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:57.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:58.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:58.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:41:59.993 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:41:59.992+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:01.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:01.017+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:02.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:02.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:03.068 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:03.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:04.097 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:04.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:05.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:05.096+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:06.066 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:06.065+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:07.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:07.109+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:08.091 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:08.089+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:09.112 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:09.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:10.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:10.151+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:11.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:11.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:12.154 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:12.152+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:13.183 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:13.181+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:14.204 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:14.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:15.040 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:42:15.040 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:42:15.040 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:42:15.040 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:15.041 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:42:15.046 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:42:15.047 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:42:15.047 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:15.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:42:15.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:15.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:16.033 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2348,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:16.065 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2348,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:16.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:16.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:17.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:17.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:18.223 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:18.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:19.066 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:19.066 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:42:19.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:19.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:20.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:20.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:21.226 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:21.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:22.268 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:22.266+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:23.291 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:23.290+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:23.919 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:23.942 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:23.942 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:42:23.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:23.943 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:42:24.324 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:24.322+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:24.925 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2354,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:42:24.948 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:24.949 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:42:25.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:25.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:26.169 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2354,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:42:26.192 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:26.192 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:42:26.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:26.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:27.203 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2354,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:42:27.228 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:42:27.228 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:27.229 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:42:27.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:27.274+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:28.278 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:28.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:28.369 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:42:29.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:29.291+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:29.370 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:42:29.370 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:29.370 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:42:30.310 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:30.308+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:30.357 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:30.379 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:30.379 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:42:30.379 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:42:30.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:30.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:42:31.297 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:31.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:31.373 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2354,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:42:31.395 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:31.395 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:42:32.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:32.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:32.398 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2354,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:42:32.421 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:32.421 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:42:33.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:33.202+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:33.414 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2354,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:42:33.444 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:42:33.444 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:42:33.444 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:42:33.444 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:42:33.444 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:42:33.445 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:42:34.194 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:34.193+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:35.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:35.187+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:36.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:36.159+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:37.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:37.146+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:38.164 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:38.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:39.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:39.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:39.546 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:42:39.546 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:42:39.547 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:39.547 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:42:40.174 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:40.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:40.528 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:40.553 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2354,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:41.143 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:41.140+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:42.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:42.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:43.100 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:43.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:43.554 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:43.554 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:42:44.098 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:44.095+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:45.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:45.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:46.096 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:46.087+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:47.058 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:47.056+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:48.042 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:48.040+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:49.048 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:49.046+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:50.062 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:50.060+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:50.767 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2358,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:50.799 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2358,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":2,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:42:50.799 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:42:50.799 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:50.799 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:42:51.046 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:51.044+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:51.792 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2358,"quorum":[0,1],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:42:51.814 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:42:51.814 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:42:52.050 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:52.048+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:52.826 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2358,"quorum":[0,1],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:42:52.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:42:53.061 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:53.059+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:54.080 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:54.078+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:55.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:55.106+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:56.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:56.117+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:57.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:57.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:58.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:58.189+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:42:59.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:42:59.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:00.230 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:00.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:01.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:01.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:02.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:02.228+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:03.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:03.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:04.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:04.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:05.200 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:05.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:06.250 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:06.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:07.284 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:07.282+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:08.282 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:08.280+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:09.325 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:09.324+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:10.279 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:10.277+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:11.231 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:11.229+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:12.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:12.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:13.239 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:13.237+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:14.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:14.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:15.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:15.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:16.188 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:16.186+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:17.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:17.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:18.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:18.204+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:19.182 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:19.180+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:20.163 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:20.161+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:21.145 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:21.143+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:22.167 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:22.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:23.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:23.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:24.151 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:24.149+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:25.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:25.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:26.148 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:26.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:27.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:27.166+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:28.169 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:28.167+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:29.217 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:29.214+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:30.237 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:30.234+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:31.258 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:31.256+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:32.221 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:32.219+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:33.216 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:33.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:34.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:34.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:35.201 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:35.199+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:36.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:36.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:37.249 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:37.247+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:38.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:38.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:39.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:39.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:40.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:40.207+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:41.234 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:41.232+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:42.218 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:42.216+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:43.224 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:43.222+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:44.207 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:44.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:45.186 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:45.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:46.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:46.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:47.187 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:47.185+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:48.160 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:48.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:49.141 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:49.139+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:50.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:50.110+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:51.161 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:51.155+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:52.202 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:52.198+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:53.225 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:53.223+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:54.185 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:54.183+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:55.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:55.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:56.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:56.142+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:57.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:57.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:58.139 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:58.137+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:43:59.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:43:59.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:00.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:00.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:01.203 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:01.201+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:02.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:02.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:03.242 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:03.240+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:04.275 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:04.273+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:05.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:05.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:06.228 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:06.226+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:07.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:07.267+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:08.311 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:08.301+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:09.335 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:09.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:10.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:10.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:11.374 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:11.372+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:12.327 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:12.325+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:13.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:13.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:14.290 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:14.288+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:15.256 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:15.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:16.300 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:16.298+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:17.266 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:17.264+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:18.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:18.254+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:19.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:19.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:20.215 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:20.212+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:21.173 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:21.170+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:22.209 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:22.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:22.849 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:44:22.850 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:44:22.850 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:44:22.850 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:22.850 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:44:22.857 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:44:22.857 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:44:22.858 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:22.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:23.227 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:23.224+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:23.883 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2358,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:23.909 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2358,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":95,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:24.261 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:24.259+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:25.269 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:25.257+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:26.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:26.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:26.912 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:26.913 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:27.294 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:27.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:28.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:28.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:29.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:29.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:29.895 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:29.918 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:29.918 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:44:29.919 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:29.919 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:44:30.265 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:30.263+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:30.918 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2364,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:44:30.943 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:30.943 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:44:31.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:31.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:32.309 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:32.307+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:33.330 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:33.329+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:34.321 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:34.317+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:34.967 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2364,"quorum":[0,1,2],"quorum_age":5,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:44:34.998 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:34.998 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:44:35.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:35.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:35.991 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2364,"quorum":[0,1,2],"quorum_age":6,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:44:36.014 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:44:36.014 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:36.014 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:44:36.385 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:36.383+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:37.116 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:44:37.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:37.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:38.117 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:44:38.117 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:38.118 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:38.436 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:38.433+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:39.126 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:39.158 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":9,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:39.159 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:44:39.159 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:44:39.159 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:39.159 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:44:39.483 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:39.480+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:40.137 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2364,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:44:40.168 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:40.168 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:44:40.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:40.510+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:41.215 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2364,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:44:41.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:41.237 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:44:41.504 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:41.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:42.246 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2364,"quorum":[0,1,2],"quorum_age":12,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:44:42.268 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2019-10-01T11:44:42.268 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:44:42.268 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2019-10-01T11:44:42.269 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:44:42.269 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-10-01T11:44:42.269 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:44:42.288 INFO:tasks.ceph.mon.c:Stopped 2019-10-01T11:44:42.288 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:44:42.288 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:42.288 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:42.461 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:42.458+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:43.356 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:43.379 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":13,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:43.470 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:43.469+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:44.432 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:44.430+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:45.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:45.390+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:46.356 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:46.354+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:46.379 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:46.380 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:47.390 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:47.405 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:47.400+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:47.421 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2364,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":17,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:48.391 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:48.389+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:49.395 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:49.393+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:50.388 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:50.386+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:50.422 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:50.422 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:44:51.433 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:51.432+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:52.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:52.412+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:53.380 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:53.378+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:54.370 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:54.368+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:55.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:55.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:56.382 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:56.381+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:56.417 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2368,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:56.450 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2368,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:44:56.450 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:44:56.450 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:56.450 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:44:57.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:57.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:57.436 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2368,"quorum":[0,1],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:44:57.469 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:44:57.470 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:44:58.316 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:58.314+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:44:58.490 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2368,"quorum":[0,1],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:44:58.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:44:59.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:44:59.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:00.293 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:00.292+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:01.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:01.252+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:02.272 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:02.270+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:03.251 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:03.248+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:04.253 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:04.251+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:05.211 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:05.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:06.168 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:06.165+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:07.191 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:07.188+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:08.144 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:08.141+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:09.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:09.132+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:10.128 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:10.123+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:11.119 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:11.116+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:12.152 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:12.150+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:13.157 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:13.154+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:14.189 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:14.179+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:15.135 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:15.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:16.121 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:16.119+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:17.099 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:17.097+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:18.107 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:18.105+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:19.065 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:19.063+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:20.111 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:20.108+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:21.134 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:21.133+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:22.149 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:22.147+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:23.195 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:23.194+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:24.170 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:24.168+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:25.136 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:25.134+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:26.165 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:26.163+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:27.150 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:27.148+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:28.171 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:28.169+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:29.179 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:29.174+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:30.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:30.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:31.254 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:31.246+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:32.281 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:32.279+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:33.308 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:33.306+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:34.343 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:34.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:35.339 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:35.338+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:36.318 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:36.316+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:37.276 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:37.275+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:38.315 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:38.313+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:39.351 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:39.349+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:40.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:40.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:41.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:41.357+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:42.349 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:42.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:43.334 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:43.332+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:44.317 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:44.315+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:45.312 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:45.311+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:46.302 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:46.300+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:47.333 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:47.331+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:48.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:48.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:49.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:49.392+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:50.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:50.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:51.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:51.376+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:52.347 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:52.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:53.402 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:53.394+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:54.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:54.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:55.344 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:55.341+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:56.359 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:56.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:57.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:57.355+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:58.366 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:58.364+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:45:59.416 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:45:59.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:00.453 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:00.450+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:01.471 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:01.468+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:02.510 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:02.509+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:03.526 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:03.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:04.545 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:04.544+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:05.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:05.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:06.574 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:06.572+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:07.603 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:07.602+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:08.564 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:08.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:09.594 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:09.592+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:10.615 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:10.613+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:11.619 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:11.617+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:12.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:12.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:13.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:13.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:14.560 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:14.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:15.584 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:15.582+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:16.567 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:16.565+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:17.571 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:17.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:18.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:18.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:19.586 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:19.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:20.556 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:20.554+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:21.593 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:21.591+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:22.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:22.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:23.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:23.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:24.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:24.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:25.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:25.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:26.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:26.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:27.605 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:27.603+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:28.514 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2019-10-01T11:46:28.514 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2019-10-01T11:46:28.514 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-01T11:46:28.515 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:28.515 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-01T11:46:28.520 INFO:tasks.ceph.mon.c:Started 2019-10-01T11:46:28.520 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:46:28.521 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:28.521 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:46:28.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:28.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:29.512 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2368,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:29.537 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2368,"quorum":[0,1],"quorum_names":["b","a"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:29.602 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:29.600+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:30.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:30.598+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:31.639 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:31.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:32.538 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:32.538 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:46:32.663 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:32.661+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:33.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:33.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:34.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:34.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:35.648 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:35.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:36.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:36.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:37.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:37.712+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:38.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:38.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:39.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:39.645+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:40.682 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:40.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:41.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:41.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:42.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:42.659+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:43.613 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:43.636 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":1,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:43.636 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:46:43.637 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:43.637 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:46:43.681 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:43.679+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:44.632 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2378,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:46:44.649 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:44.646+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:44.654 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:44.655 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:46:45.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:45.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:45.735 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2378,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:46:45.766 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:45.766 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:46:46.613 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:46.610+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:46.755 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2378,"quorum":[0,1,2],"quorum_age":4,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:46:46.777 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:46:46.777 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:46.777 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:46:47.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:47.650+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:47.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:46:48.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:48.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:48.862 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:46:48.862 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:48.863 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:46:49.655 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:49.652+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:49.865 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:49.887 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":7,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:49.888 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:46:49.888 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:46:49.888 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:49.888 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:46:50.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:50.683+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:50.872 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2378,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:46:50.894 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:50.895 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:46:51.650 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:51.648+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:51.882 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2378,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:46:51.910 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:51.910 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:46:52.669 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:52.665+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:52.904 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2378,"quorum":[0,1,2],"quorum_age":11,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:46:52.930 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:46:52.930 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:46:52.930 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:46:52.930 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:46:52.930 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:46:52.930 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:46:52.951 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:46:52.951 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:46:52.951 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:52.951 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:46:53.683 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:53.681+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:53.933 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:53.964 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":12,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:54.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:54.641+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:55.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:55.691+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:56.668 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:56.666+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:56.965 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:46:56.965 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:46:57.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:57.676+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:57.944 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:57.969 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2378,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":16,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:46:58.629 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:58.627+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:46:59.646 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:46:59.643+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:00.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:00.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:00.970 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:47:00.970 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:47:01.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:01.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:02.597 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:02.593+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:03.598 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:03.596+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:04.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:04.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:05.620 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:05.618+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:06.622 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:06.620+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:07.582 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:07.580+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:08.021 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:47:08.051 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:47:08.052 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:47:08.052 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:47:08.052 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:47:08.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:08.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:09.023 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2380,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:47:09.046 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:47:09.047 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:47:09.581 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:09.579+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:10.042 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2380,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:47:10.073 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:47:10.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:10.538+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:11.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:11.577+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:12.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:12.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:13.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:13.534+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:14.570 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:14.568+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:15.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:15.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:16.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:16.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:17.578 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:17.576+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:18.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:18.588+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:19.553 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:19.552+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:20.583 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:20.581+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:21.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:21.614+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:22.664 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:22.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:23.686 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:23.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:24.637 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:24.635+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:25.687 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:25.685+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:26.685 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:26.680+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:27.634 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:27.632+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:28.596 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:28.594+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:29.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:29.548+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:30.561 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:30.559+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:31.609 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:31.584+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:32.600 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:32.597+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:33.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:33.556+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:34.558 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:34.555+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:35.544 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:35.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:36.590 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:36.587+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:37.612 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:37.609+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:38.627 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:38.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:39.665 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:39.662+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:40.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:40.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:41.633 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:41.631+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:42.607 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:42.605+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:43.644 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:43.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:44.638 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:44.636+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:45.626 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:45.624+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:46.661 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:46.660+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:47.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:47.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:48.653 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:48.651+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:49.666 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:49.664+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:50.643 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:50.642+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:51.616 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:51.601+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:52.642 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:52.640+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:53.654 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:53.653+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:54.696 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:54.694+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:55.679 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:55.677+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:56.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:56.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:57.763 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:57.761+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:58.724 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:58.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:47:59.695 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:47:59.693+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:00.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:00.737+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:01.750 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:01.748+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:02.777 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:02.775+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:03.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:03.800+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:04.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:04.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:05.748 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:05.746+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:06.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:06.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:07.751 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:07.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:08.720 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:08.718+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:09.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:09.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:10.722 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:10.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:11.706 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:11.687+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:12.712 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:12.710+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:13.753 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:13.751+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:14.765 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:14.764+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:15.797 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:15.795+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:16.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:16.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:17.785 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:17.783+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:18.801 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:18.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:19.807 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:19.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:20.833 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:20.831+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:21.830 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:21.828+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:22.813 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:22.810+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:23.855 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:23.853+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:24.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:24.882+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:25.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:25.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:26.879 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:26.877+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:27.852 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:27.849+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:28.902 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:28.899+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:29.891 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:29.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:30.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:30.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:31.990 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:31.988+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:32.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:32.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:33.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:33.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:34.949 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:34.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:35.912 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:35.910+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:36.942 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:36.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:37.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:37.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:38.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:38.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:39.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:39.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:40.074 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:48:40.074 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2019-10-01T11:48:40.074 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-01T11:48:40.075 INFO:teuthology.orchestra.run.mira064:Running: 2019-10-01T11:48:40.075 INFO:teuthology.orchestra.run.mira064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-01T11:48:40.088 INFO:tasks.ceph.mon.b:Started 2019-10-01T11:48:40.088 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:48:40.088 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:48:40.089 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:48:40.960 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:40.958+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:41.091 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:41.122 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:41.951 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:41.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:42.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:42.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:43.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:43.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:44.123 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:48:44.123 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:48:44.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:44.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:45.122 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:45.144 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:45.926 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:45.924+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:46.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:46.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:47.945 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:47.943+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:48.145 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:48:48.145 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:48:48.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:48.953+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:49.160 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:49.182 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:49.944 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:49.942+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:50.995 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:50.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:52.035 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:52.033+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:52.183 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:48:52.184 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:48:53.040 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:53.038+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:53.179 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:53.210 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2380,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":105,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:48:54.084 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:54.077+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:55.059 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:55.057+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:56.052 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:56.051+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:56.211 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:48:56.211 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:48:57.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:57.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:58.043 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:58.042+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:48:59.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:48:59.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:00.009 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:00.007+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:00.992 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:00.985+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:01.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:01.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:02.997 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:02.995+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:03.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:03.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:04.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:04.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:05.972 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:05.957+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:06.519 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:06.545 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:06.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:49:06.546 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:06.546 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:49:06.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:06.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:07.647 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2394,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:07.665 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:07.665 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:49:07.952 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:07.950+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:08.628 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2394,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:08.659 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:08.660 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:49:08.924 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:08.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:09.661 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2394,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:09.683 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:49:09.684 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:09.684 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:49:09.894 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:09.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:10.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:49:10.920 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:10.918+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:11.851 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:49:11.851 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:11.851 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:49:11.882 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:11.880+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:12.864 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:12.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:12.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:12.896 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:12.897 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:49:12.897 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:49:12.897 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:12.897 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:49:13.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:13.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:13.984 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2394,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:14.016 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:14.016 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:49:14.869 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:14.866+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:15.034 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2394,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:15.058 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:15.058 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:49:15.827 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:15.825+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:16.062 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2394,"quorum":[0,1,2],"quorum_age":10,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:16.287 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2019-10-01T11:49:16.288 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:49:16.288 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2019-10-01T11:49:16.288 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:49:16.288 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-10-01T11:49:16.288 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:49:16.329 INFO:tasks.ceph.mon.a:Stopped 2019-10-01T11:49:16.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:49:16.329 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:16.330 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:49:16.794 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:16.791+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:17.344 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:17.367 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:17.798 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:17.796+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:18.752 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:18.749+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:19.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:19.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:20.368 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:20.368 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:49:20.770 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:20.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:21.367 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:21.390 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:21.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:21.736+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:22.739 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:22.735+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:23.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:23.722+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:24.391 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:24.392 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:49:24.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:24.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:25.410 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:25.440 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2394,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:25.749 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:25.747+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:26.744 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:26.742+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:27.715 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:27.714+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:28.440 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:28.441 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:49:28.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:28.725+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:29.710 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:29.708+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:30.732 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:30.724+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:31.180 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2398,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:31.208 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2398,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:49:31.208 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:49:31.208 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:31.208 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:49:31.713 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:31.711+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:32.204 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2398,"quorum":[0,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:49:32.227 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:49:32.228 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:49:32.756 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:32.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:33.225 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2398,"quorum":[0,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:49:33.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:49:33.793 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:33.790+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:34.836 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:34.834+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:35.873 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:35.871+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:36.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:36.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:37.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:37.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:38.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:38.927+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:39.915 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:39.913+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:40.948 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:40.946+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:41.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:41.948+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:42.981 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:42.979+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:44.016 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:44.014+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:45.007 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:45.005+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:46.011 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:46.009+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:46.967 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:46.966+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:47.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:47.939+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:48.947 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:48.945+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:49.918 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:49.916+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:50.885 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:50.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:51.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:51.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:52.883 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:52.881+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:53.903 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:53.902+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:54.893 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:54.892+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:55.876 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:55.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:56.864 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:56.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:57.838 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:57.836+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:58.825 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:58.823+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:49:59.840 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:49:59.839+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:00.842 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:00.841+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:01.805 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:01.801+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:02.848 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:02.845+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:03.871 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:03.862+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:04.817 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:04.814+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:05.772 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:05.769+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:06.796 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:06.793+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:07.773 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:07.770+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:08.757 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:08.755+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:09.708 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:09.707+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:10.745 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:10.743+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:11.726 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:11.723+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:12.677 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:12.675+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:13.721 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:13.719+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:14.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:14.733+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:15.733 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:15.731+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:16.742 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:16.740+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:17.747 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:17.745+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:18.703 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:18.701+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:19.701 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:19.699+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:20.693 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:20.686+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:21.707 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:21.705+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:22.754 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:22.752+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:23.718 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:23.716+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:24.734 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:24.732+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:25.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:25.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:26.723 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:26.720+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:27.719 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:27.717+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:28.740 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:28.738+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:29.705 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:29.703+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:30.735 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:30.734+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:31.764 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:31.762+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:32.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:32.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:33.803 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:33.799+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:34.821 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:34.819+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:35.787 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:35.785+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:36.762 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:36.759+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:37.774 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:37.772+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:38.820 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:38.818+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:39.857 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:39.855+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:40.865 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:40.864+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:41.890 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:41.889+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:42.907 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:42.905+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:43.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:43.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:44.911 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:44.909+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:45.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:45.876+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:46.895 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:46.893+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:47.884 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:47.883+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:48.880 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:48.878+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:49.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:49.914+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:50.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:50.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:51.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:51.922+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:52.878 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:52.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:53.901 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:53.898+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:54.877 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:54.874+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:55.925 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:55.919+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:56.950 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:56.947+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:57.935 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:57.933+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:58.917 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:58.915+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:50:59.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:50:59.908+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:00.954 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:00.952+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:01.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:01.930+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:02.928 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:02.926+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:03.257 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2019-10-01T11:51:03.257 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2019-10-01T11:51:03.257 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-01T11:51:03.258 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:03.258 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-01T11:51:03.268 INFO:tasks.ceph.mon.a:Started 2019-10-01T11:51:03.268 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:51:03.268 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:03.269 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:03.914 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:03.912+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:04.242 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2398,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:04.264 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2398,"quorum":[0,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":93,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:04.910 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:04.906+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:05.932 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:05.931+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:06.974 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:06.972+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:07.265 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:07.265 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:07.946 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:07.944+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:08.999 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:08.993+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:10.003 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:10.001+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:11.051 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:11.049+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:12.038 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:12.036+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:12.808 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:12.830 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:12.830 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:51:12.831 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:12.831 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:51:13.056 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:13.054+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:13.851 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2406,"quorum":[0,1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:13.873 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:13.874 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:51:14.019 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:14.016+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:14.833 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2406,"quorum":[0,1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:51:14.857 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:14.858 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:51:15.049 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:15.047+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:15.960 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2406,"quorum":[0,1,2],"quorum_age":3,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:15.979 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2019-10-01T11:51:15.979 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:15.979 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph scrub 2019-10-01T11:51:16.067 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:16.066+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:17.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2019-10-01T11:51:17.116 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:17.114+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:18.102 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2019-10-01T11:51:18.103 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:18.103 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:18.155 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:18.153+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:19.075 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:19.096 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":6,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:19.096 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2019-10-01T11:51:19.096 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2019-10-01T11:51:19.097 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:19.097 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:51:19.172 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:19.171+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:20.109 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"peon","election_epoch":2406,"quorum":[0,1,2],"quorum_age":7,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:20.138 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:20.139 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:51:20.193 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:20.192+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:21.103 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2406,"quorum":[0,1,2],"quorum_age":8,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}]}} 2019-10-01T11:51:21.125 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:21.126 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.120 mon_status 2019-10-01T11:51:21.205 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:21.203+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:22.129 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"b","rank":0,"state":"leader","election_epoch":2406,"quorum":[0,1,2],"quorum_age":9,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:22.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2019-10-01T11:51:22.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2019-10-01T11:51:22.147 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2019-10-01T11:51:22.147 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2019-10-01T11:51:22.148 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-10-01T11:51:22.148 INFO:teuthology.orchestra.run:waiting for 300 2019-10-01T11:51:22.171 INFO:tasks.ceph.mon.b:Stopped 2019-10-01T11:51:22.171 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2019-10-01T11:51:22.173 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:22.173 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:22.198 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:22.196+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:23.185 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:23.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:23.205+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:23.208 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":11,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:24.214 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:24.213+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:25.213 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:25.211+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:26.208 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:26.206+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:26.209 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:26.209 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:27.214 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:27.236 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":15,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:27.257 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:27.255+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:28.301 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:28.299+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:29.307 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:29.295+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:30.237 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:30.238 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:30.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:30.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:31.231 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:31.254 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2406,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","quorum_age":19,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:31.364 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:31.362+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:32.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:32.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:33.417 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:33.415+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:34.255 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:34.256 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2019-10-01T11:51:34.393 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:34.391+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:35.415 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:35.413+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:36.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:36.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:37.325 INFO:teuthology.orchestra.run.mira063.stdout:{"election_epoch":2408,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:37.348 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":2408,"quorum":[1,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]}} 2019-10-01T11:51:37.348 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2019-10-01T11:51:37.348 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:37.348 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m 172.21.6.124 mon_status 2019-10-01T11:51:37.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:37.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:38.367 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"a","rank":1,"state":"leader","election_epoch":2408,"quorum":[1,2],"quorum_age":1,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:38.391 INFO:teuthology.orchestra.run.mira063:Running: 2019-10-01T11:51:38.391 INFO:teuthology.orchestra.run.mira063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -m '[v2:172.21.6.124:3301,v1:172.21.6.124:6790]' mon_status 2019-10-01T11:51:38.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:38.523+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:39.389 INFO:teuthology.orchestra.run.mira063.stdout:{"name":"c","rank":2,"state":"peon","election_epoch":2408,"quorum":[1,2],"quorum_age":2,"features":{"required_con":"2449958747315978244","required_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"outside_quorum":[],"extra_probe_peers":[],"sync_provider":[],"monmap":{"epoch":1,"fsid":"3fb5d1e9-4bed-4c52-a6cc-37e3dc1ef525","modified":"2019-10-01T02:22:51.990140Z","created":"2019-10-01T02:22:51.990140Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.120:3300","nonce":0},{"type":"v1","addr":"172.21.6.120:6789","nonce":0}]},"addr":"172.21.6.120:6789/0","public_addr":"172.21.6.120:6789/0","priority":0,"weight":0},{"rank":1,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3300","nonce":0},{"type":"v1","addr":"172.21.6.124:6789","nonce":0}]},"addr":"172.21.6.124:6789/0","public_addr":"172.21.6.124:6789/0","priority":0,"weight":0},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.6.124:3301","nonce":0},{"type":"v1","addr":"172.21.6.124:6790","nonce":0}]},"addr":"172.21.6.124:6790/0","public_addr":"172.21.6.124:6790/0","priority":0,"weight":0}]},"feature_map":{"mon":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}],"osd":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":4}],"client":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":2}],"mgr":[{"features":"0x3f01cfb8ffadffff","release":"luminous","num":1}]}} 2019-10-01T11:51:39.412 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2019-10-01T11:51:39.540 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:39.536+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:40.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:40.499+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:41.488 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:41.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:42.538 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:42.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:43.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:43.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:44.519 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:44.516+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:45.565 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:45.562+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:46.541 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:46.539+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:47.550 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:47.549+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:48.524 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:48.522+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:49.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:49.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:50.525 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:50.524+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:51.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:51.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:52.521 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:52.519+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:53.536 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:53.515+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:54.487 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:54.485+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:55.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:55.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:56.492 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:56.490+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:57.528 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:57.526+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:58.494 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:58.492+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:51:59.505 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:51:59.503+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:00.534 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:00.532+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:01.548 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:01.542+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:02.549 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:02.547+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:03.559 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:03.557+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:04.543 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:04.541+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:05.530 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:05.528+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:06.573 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:06.571+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:07.562 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:07.560+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:08.515 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:08.513+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:09.502 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:09.500+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:10.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:10.531+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:11.539 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:11.537+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:12.537 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:12.535+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:13.532 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:13.530+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:14.500 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:14.498+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:15.451 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:15.449+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:16.423 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:16.421+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:17.424 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:17.422+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:18.458 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:18.456+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:19.469 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:19.467+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:20.454 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:20.452+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:21.410 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:21.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:22.411 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:22.408+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:23.361 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:23.359+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:24.377 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:24.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:25.358 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:25.356+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:26.348 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:26.346+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:27.350 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:27.348+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:28.363 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:28.361+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:29.408 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:29.405+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:30.378 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:30.375+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:31.407 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:31.404+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:32.387 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:32.385+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:33.346 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:33.343+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:34.345 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:34.342+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:35.336 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:35.334+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:36.376 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:36.374+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:37.372 INFO:tasks.ceph.osd.0.mira063.stderr:2019-10-01T11:52:37.370+0000 7f10b36bc700 -1 osd.0 38 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4386.0:335 2.a 2.1cd3cdda (undecoded) ondisk+retry+write+known_if_redirected e37) 2019-10-01T11:52:37.946 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-01T11:52:37.946 DEBUG:teuthology.task.console_log:Killing console logger for mira063 2019-10-01T11:52:37.946 DEBUG:teuthology.task.console_log:Killing console logger for mira064 2019-10-01T11:52:37.947 DEBUG:teuthology.task.console_log:Killing console logger for mira063 2019-10-01T11:52:37.947 DEBUG:teuthology.task.console_log:Killing console logger for mira064 2019-10-01T11:52:37.948 DEBUG:teuthology.exit:Finished running handlers